Simplified user interface generation

ABSTRACT

Methods, systems, devices, and tangible non-transitory computer readable media for route planning are provided. The disclosed technology can include accessing user interface data that includes states of interface elements associated with a user interface. Further, the state of the user interface can be determined based at least in part on the user interface data. The state of the user interface can include functionalities of the interface elements. The interface elements can be mapped to simplified user interface elements on the based at least in part on the functionalities of the interface elements. Furthermore, a simplified user interface can be generated based at least in part on the simplified user interface elements. The simplified user interface elements can be configured to perform operations associated with the functionalities of the interface elements.

FIELD

The present disclosure relates generally to the generation of userinterfaces. More particularly, the present disclosure relates togenerating simplified user interfaces.

BACKGROUND

Applications, including software applications, can be implemented on avariety of computing devices (e.g., smartphones, tablet computingdevices, or wearable computing devices). These applications can performa variety of functions and can include a user interface that allows auser to interact with the application and perform operations includingcontrolling certain aspects of the application. However, applicationsand the way in which the applications are used can change over time,which may create a need for different ways of interacting with theapplications. Further, certain users may find certain user interfacesconfusing, cluttered, or otherwise too complex to permit intuitive,convenient, and/or useful interaction between the user and theinterface. Accordingly, there exists a demand for a more effective wayof interacting with an application, and in particular a need for a moreeffective user interface that facilitates user interaction with theapplication.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to acomputer-implemented method of user interface generation. Thecomputer-implemented method can include accessing, by a user computingdevice associated with a user and comprising one or more processors,user interface data comprising one or more states of a user interfacecomprising one or more interface elements. The computer-implementedmethod can include determining, by the user computing device, a state ofthe user interface based at least in part on the user interface data.The state of the user interface can include one or more functionalitiesof the one or more interface elements. The computer-implemented methodcan include mapping, by the user computing device, the one or moreinterface elements to one or more simplified user interface elementsbased at least in part on the one or more functionalities of the one ormore interface elements. Furthermore, the computer-implemented methodcan include generating, by the user computing device, a simplified userinterface based at least in part on the one or more simplified userinterface elements. The one or more simplified user interface elementscan be configured to perform one or more operations associated with theone or more functionalities of the one or more interface elements.

Another example aspect of the present disclosure is directed to one ormore tangible non-transitory computer-readable media storingcomputer-readable instructions that when executed by one or moreprocessors cause the one or more processors to perform operations. Theoperations can include accessing user interface data comprising one ormore states of a user interface comprising one or more interfaceelements. The operations can include determining, by the user computingdevice, a state of the user interface based at least in part on the userinterface data. The state of the user interface can include one or morefunctionalities of the one or more interface elements. The operationscan include mapping, by the user computing device, the one or moreinterface elements to one or more simplified user interface elementsbased at least in part on the one or more functionalities of the one ormore interface elements. Furthermore, the operations can includegenerating, by the user computing device, a simplified user interfacebased at least in part on the one or more simplified user interfaceelements. The one or more simplified user interface elements can beconfigured to perform one or more operations associated with the one ormore functionalities of the one or more interface elements.

Another example aspect of the present disclosure is directed to acomputing system comprising: one or more processors; one or morenon-transitory computer-readable media storing instructions that whenexecuted by the one or more processors cause the one or more processorsto perform operations. The operations can include accessing userinterface data comprising one or more states of a user interfacecomprising one or more interface elements. The operations can includedetermining, by the user computing device, a state of the user interfacebased at least in part on the user interface data. The state of the userinterface can include one or more functionalities of the one or moreinterface elements. The operations can include mapping, by the usercomputing device, the one or more interface elements to one or moresimplified user interface elements based at least in part on the one ormore functionalities of the one or more interface elements. Furthermore,the operations can include generating, by the user computing device, asimplified user interface based at least in part on the one or moresimplified user interface elements. The one or more simplified userinterface elements can be configured to perform one or more operationsassociated with the one or more functionalities of the one or moreinterface elements.

Other aspects of the present disclosure are directed to various systems,apparatuses, non-transitory computer-readable media, user interfaces,and electronic devices.

These and other features, aspects, and advantages of various embodimentsof the present disclosure will become better understood with referenceto the following description and appended claims. The accompanyingdrawings, which are incorporated in and constitute a part of thisspecification, illustrate example embodiments of the present disclosureand, together with the description, serve to explain the relatedprinciples.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1A depicts a block diagram of an example computing system thatperforms operations associated with the generation of a simplified userinterface according to example embodiments of the present disclosure.

FIG. 1B depicts a block diagram of an example computing device thatperforms operations associated with the generation of a simplified userinterface according to example embodiments of the present disclosure.

FIG. 1C depicts a block diagram of an example computing device thatperforms operations associated with the generation of a simplified userinterface according to example embodiments of the present disclosure.

FIG. 2 depicts a block diagram of an example of one or moremachine-learned models according to example embodiments of the presentdisclosure.

FIG. 3 depicts an example of a user computing device according toexample embodiments of the present disclosure;

FIG. 4 depicts an example of user interaction with a user computingdevice according to example embodiments of the present disclosure;

FIG. 5 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 6 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 7 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 8 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 9 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 10 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 11 depicts an example of a simplified user interface according toexample embodiments of the present disclosure;

FIG. 12 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure;

FIG. 13 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure;

FIG. 14 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure; and

FIG. 15 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure.

Reference numerals that are repeated across plural figures are intendedto identify the same features in various implementations.

DETAILED DESCRIPTION

Generally, the present disclosure is directed to the generation of userinterfaces including simplified user interfaces. Example aspects of thepresent disclosure are directed to a simplified user interfacegeneration system that can be used to generate a simplified userinterface based on the state of an existing user interface. In someimplementations, the simplified user interface generation system can belocated on and executed locally by a user's computing device (e.g.,smartphone) to augment or replace user interfaces presented byapplication(s) on the user's computing device. Additionally oralternatively, the simplified user interface generation system can belocated at a server computing system and can serve the simplified userinterfaces to client devices such as user devices. According to anotheraspect, in some implementations, the simplified user interfacegeneration system can include and leverage one or more machine-learnedmodels (e.g., “on-device” machine learned models) to generate thesimplified user interface(s). For example, a machine-learned model canlearn to map current state(s) and/or element(s) of an existing userinterface to state(s) and/or element(s) (e.g., a reduced number ofelements) of a simplified user interface.

In particular, the disclosed technology can be used to determine thestate of a user interface and generate a simplified user interface thatcan perform tasks associated with the interface elements of the userinterface. For example, a simplified user interface can be generatedthat reduces the number of interface elements with which a user willinteract. Further, the interface elements that make up the simplifieduser interface can be determined based on which interface elements aremost relevant within the context of the user interface.

The simplified interface can be presented to the user. For example, thesimplified interface can be presented in addition to and/or incorrespondence with the existing user interface such as, e.g., overlaidupon the existing interface. Alternatively, the simplified userinterface can replace the existing user interface. In one example, thesimplified interface can be an “action bar” that is presented (e.g., ata bottom of a current display screen) and which includes the mostimportant control options for a user. As such, the disclosed technologycan ensure that a user is provided with a simplified user interface thatcan facilitate user interactions to perform tasks in an application.

The disclosed technology can be implemented in a computing system orcomputing device (e.g., a user computing device) that is configured toaccess data, perform operations on the data (e.g., determining the stateof a user interface and map user interface elements to simplified userinterface elements), and generate output including a simplified userinterface. Further, the computing system can be included in a computingdevice (e.g., a personal computing device including a smartphone) or aspart of a system that includes multiple devices that can communicateand/or interact with one another.

By way of example, the disclosed technology can include a user computingdevice that accesses user interface data that includes informationassociated with the state of a user interface. The user interface can bepresented to the user in a visual medium such as a graphical userinterface that is displayed on a display device. Further, the usercomputing device can determine the state of the user interface based onthe user interface data which can include information associated withthe inputs and outputs that the user interface is respectively able toreceive and generate. For example, the user interface can receive touchinputs via a touch-screen display and generate visual and auditoryoutputs in response to the touch inputs. The user computing can then mapinterface elements included in the user interface (e.g., controlelements that a user can use to activate applications) to simplifieduser interface elements that perform the tasks of their correspondinginterface elements and which can be presented in a form that is simplerand easier to interact with. The user computing device can then generatea simplified user interface that is based on the simplified userinterface elements and which can be used to perform tasks associatedwith the user interface. In this way, the disclosed technology canpresent the user with a simplified user interface that facilitates theuser's interactions with the computing device and allows the user tomore effectively perform tasks.

The simplified user interface generated by the disclosed technology caninclude a subset of interface elements (e.g., interface controls used toprovide input to the user interface and/or application) of the userinterface that are presented in a standardized and consistent way,thereby facilitating user interactions with the user interface. Inaddition to these interface elements, the simplified user interface canalso provide a simplified interface to communicate task-specific inputsand outputs between the user and the software.

In some implementations of the disclosed technology, a simplified userinterface can be based at least in part on a simplified user interfacecreation service that operates in the background of a computing device'soperating system and monitors the state of the user interface. Wheneverthe state of the user interface changes, the simplified user interfacecreation service can determine the type and/or class of the userinterface that is being presented and selects an appropriate simplifieduser interface template based on the type or class. Further, an actionprediction service can analyze the user interface and rank differentelements based on how likely those elements are to be activated by auser. Further, an interface element understanding service can assign aclass to the interface elements and determine a corresponding simplifieduser interface element (e.g., a control represented by an onscreen icon)representative icon for the respective interface elements. Thesimplified user interface can then be generated based on thesesimplified user interface elements.

The disclosed technology can include a user interface state detector,that can determine and/or detect whether the current user interfacestate is the same state as the user interface state for which asimplified user interface configuration was stored. On a positivedetermination, the appropriate simplified user interface can begenerated. Further, the disclosed technology can include an interfaceelement mapper that can map a simplified user interface element on thesimplified user interface to an interface element on the (un-simplified)user interface.

Accordingly, the disclosed technology can improve the ergonomics andefficiency of user interaction with an interface by providing a userwith a simplified user interface that can be generated based on avariety of factors including a user's probable interactions with theinterface and the current layout of the interface. Further, thedisclosed technology can assist a user in more effectively performingthe technical task of interacting with a user interface to facilitatethe performance of tasks that are controlled via the user interface byway of a continued and/or guided human-machine interaction process.

The user computing device can access, receive, retrieve, and/or obtaindata including user interface data. The user interface data can includeone or more states of a user interface that includes one or moreinterface elements. The user interface data can include one or morefeatures and/or one or more attributes of the user interface includingone or more features and/or one or more attributes of the one or moreinterface elements. For example, the one or more features of the userinterface can include per-pixel information including the color andlocation of each pixel of a plurality of pixels used to display the userinterface. By way of further example, the one or more attributes of theuser interface can include a layout of the one or more interfaceelements, which in some embodiments can be included as part of a treehierarchy or node hierarchy that describes one or more relationshipsbetween the one or more interface elements.

The user computing device can determine a state of the user interfacebased at least in part on the user interface data. Further, the usercomputing device can use the user interface data to determineinformation that is associated with one or more inputs that can beprovided to the user interface (e.g., voice inputs and/or tactileinputs), one or more outputs (e.g., one or more visual outputs and/orauditory outputs) of the user interface, one or more operations, and/orone or more tasks (e.g., one or more information retrieval tasks,application operation tasks (e.g., opening and/or closing applications),and/or communications tasks (e.g., sending information to and/orreceiving information from an application)).

The state of the user interface can include one or more functionalitiesof the user interface. The one or more functionalities of the userinterface can include one or more inputs that the user interface canreceive (e.g., tactile interactions by a user with an interface elementand/or data received by the user interface from an application); one ormore outputs that the user interface can generate (e.g., visual outputsto a display portion of the user interface and/or data sent to anapplication by the user interface); and/or one or more operations thatcan be performed by one or more applications associated with the userinterface. Further, the one or more inputs can include one or moretask-directed inputs and/or one or more task-auxiliary inputs; and theone or more outputs can include one or more task-directed outputs and/orone or more task-auxiliary outputs.

The one or more task-directed inputs and/or the one or moretask-directed outputs can be associated with one or more tasks that areextrinsic to user interaction with the user interface. In someembodiments, the one or more task-directed inputs and/or the one or moretask-directed outputs can be intrinsic to the one or more tasks. Forexample, the one or more task-directed outputs can include one or moreoutputs such as showing a visual representation of a destinationlocation in a mapping and routing application. Further, the one or moretask-directed inputs and/or the one or more task-directed outputs canperform functions and/or operations that are required to perform a task.For example, a task directed input can include receiving an inputindicating a destination to which a user would like to travel from theircurrent location.

The one or more task-auxiliary inputs and/or the one or moretask-auxiliary outputs can be associated with one or more tasks that arenot extrinsic to user interaction with the user interface. Further, theone or more task-auxiliary inputs and/or the one or more task-auxiliaryoutputs can perform functions and/or operations that are not required toperform a task and that may be directed to interaction with the userinterface itself. For example, a task-auxiliary input can includechanging from earpiece to speaker mode via the user interface of atelephone application.

In some embodiments, the user interface can be configured to receive oneor more task-directed inputs and/or one or more task-auxiliary inputs.For example, the user interface can receive one or more task-directedinputs comprising information associated with the location of the userdevice for a user interface of a mapping application. By way of furtherexample, the user interface can receive one or more task-auxiliaryinputs comprising a user interaction to a map magnification interfaceelement that is used to control the magnification (e.g., zoom level) fora user interface of the mapping application.

Further, the user interface can be configured to generate one or moretask-directed outputs and/or one or more task-auxiliary outputs. Forexample, the user interface can generate one or more task-directedoutputs comprising audio indicating the location of the user device fora user interface of a mapping application. By way of further example,the user interface can generate one or more task-auxiliary outputscomprising displaying a magnified image of a map in response toreceiving a user input to zoom into the map of the user interface of themapping application.

In some embodiments, the user computing device can determine the stateof the user interface based at least in part on one or more features ofthe user interface. For example, the user computing device can analyzethe user interface data to determine the one or more features of the oneor more interface elements. The one or more features of the one or moreinterface elements can then be used to determine the appearance and/orfunctionality of the one or more interface elements respectively.

The one or more features of the one or more interface elements caninclude one or more text labels associated with the one or moreinterface elements (e.g., an interface element associated with a mappingapplication can be associated with a text label indicating a location),a visual appearance of the one or more interface elements (e.g., one ormore colors, one or more shapes, and/or one or more patterns associatedwith the one or more interface elements), one or more sizes of the oneor more interface elements (e.g., the size of the one or more interfaceelements relative to one another or in relation to the user interface),one or more locations of the one or more interface elements (e.g., therelative location of the one or more interface elements relative to oneanother; or the absolute location of the one or more interface elementsin relation to the user interface), one or more input modalities of theone or more interface elements (e.g., the manner or way in which the oneor more interface elements receive input including user input), and/orone or more output modalities of the one or more interface elements(e.g., the manner or way in which the one or more interface elementsprovide output including visual output and/or auditory output).

By way of example, the one or more input modalities can include tactileinput modalities (e.g., a touch screen display or a button), visualinput modalities (e.g., a camera), and/or auditory input modalities(e.g., a microphone). Further, the one or more output modalities caninclude tactile output modalities (e.g., haptic feedback generated by avibration motor), visual output modalities (e.g., a display device),and/or auditory output modalities (e.g., a loudspeaker).

In some embodiments, determining the state of the user interface can bebased at least in part on one more machine-learned models and the userinterface data. The one or more machine-learned models can be configuredand/or trained to classify the user interface, the user interface data,and/or the one or more interface elements into one or more classes(e.g., one or more semantic classes) respectively. For example, the oneor more machine-learned models can be configured and/or trained toreceive an input including the user interface and/or the user interfacedata and generate an output including one or more classes that areassociated with the user interface. For example, the one or moremachine-learned models can classify a user interface as being aparticular type of application (e.g., an text messaging application),within a particular context (e.g., the compose text message screen), anda particular set of interface elements (e.g., send interface element,compose interface element, attachment interface element). As such, thestate of the user interface can be associated with the one or moreclasses that are associated with the user interface. Further, the stateof the user interface can be associated with a simplified user interfacethat satisfies one or more criteria with respect to the similarity ofthe one or more classes of the user interface and the one or moreclasses of the simplified user interface.

Furthermore, the one or more machine-learned models can be configuredand/or trained using one or more supervised learning techniques and/orunsupervised learning techniques. Further, the one or moremachine-learned models can be configured and/or trained using trainingdata that includes one or more images of user interfaces, one or moreimages of user interface elements, and/or information associated withone or more attributes of the user interfaces or user interfaceelements. For example, the training data can include screen captures ofuser interfaces and/or user interface elements. Further, the userinterface and/or the user interface elements can be tagged to identifyone or more functionalities respectively. Based at least in part oninput including the training data, the one or more machine-learnedmodels can be configured and/or trained to recognize, detect, and/oridentify one or more visual features of the images and thereby determinethe state of the user interfaces depicted in the training data.

After being configured and/or trained, the one or more machine-learnedmodels can receive input including the user interface data; perform oneor more operations associated with the recognition, detection, and/oridentification of one or more features of the user interface data; andgenerate an output including one or more semantic classes. The one ormore semantic classes can be associated with one or more interfaceelements and/or one or more simplified user interface elements. Forexample, the one or more semantic classes can identify one or morefunctionalities of the one or more interface elements and can includeinformation associated with one or more types of input that can bereceived by the one or more interface elements, one or more types ofoutput that can be generated by the one or more interface elements, oneor more applications that are associated with the one or more interfaceelements, and/or one or more types of information that are associatedwith the one or more interface elements. Furthermore, each of the one ormore semantic classes can be associated with a set of the one or moresimplified user interface elements, thereby facilitating mapping of theone or more interface elements to the one or more simplified userinterface elements based on the one or more semantic classes that arethe same in the one or more interface elements and the one or moresimplified user interface elements.

In some embodiments, the one or more interface elements and/or the oneor more simplified user interface elements are clustered into the one ormore semantic classes based at least in part on one or more features ofthe one or more interface elements. For example, a clustering algorithm(e.g., k-means clustering or mean shift clustering) can be used todetermine the clusters of previously recorded user interface statesbased on the one or more features of the respective user interfacestates.

The user computing device can map the user interface to a simplifieduser interface. Further, the user computing device can map the one ormore interface elements to one or more simplified user interfaceelements. Mapping the one or more interface elements to the one or moresimplified user interface elements can be based at least in part on theone or more functionalities of the one or more interface elements. Forexample, the one or more interface elements for the user interface of ane-mail application can be associated with a composition functionality tocompose a new e-mail, a send functionality to send an e-mail, and acancel functionality to close a currently active window with which auser is interacting. The user computing device can compare the one ormore functionalities of the one or more interface elements of the e-mailapplication to one or more simplified user interface elements thatperform similar (or the same) functions as the respective one or moreinterface elements. When an interface element is determined tocorrespond to a simplified user interface element, the interface elementcan be mapped to the simplified user interface element.

In some embodiments, mapping the user interface to the simplified userinterface can include one or more comparisons of the user interface toone or more previously recorded user interfaces and/or one or moresimplified user interfaces. The one or more previously recorded userinterfaces can be based at least in part on analysis of one or morepreviously recorded user interfaces in which the state of eachpreviously recorded user interface and the one or more functionalitiesassociated with one or more interface elements of the one or morepreviously recorded user interfaces was stored.

The one or more previously recorded user interfaces and/or the one ormore simplified user interfaces can be classified according to theirrespective one or more features. For example, one or moremachine-learned models can be used to classify the one or morepreviously recorded user interfaces based on their respective one ormore features. Further, one or more machine-learned models can be usedto classify the one or more simplified user interfaces based on theirrespective features. Each of the one or more previously recorded userinterfaces can then be mapped to a respective one of the one or moresimplified user interfaces based at least in part on their similarity(e.g., membership in one or more common classes). Further, one or moremachine-learned models can be used to classify the user interface. Theuser interface can then be compared to the one or more previouslyrecorded user interfaces to determine the previously recorded userinterface that is most similar (e.g., that has the most classes incommon). The simplified user interface associated with the previouslyrecorded user interface that is most similar to the user interface canthen be mapped to the user interface.

In some embodiments, the user interface data can include a treehierarchy and/or a node hierarchy that is associated with the one ormore interface elements. For example, each of the one or more interfaceelements can be associated with a respective node in a node hierarchy.Further, the relative position of each node in the node hierarchy and/orthe associated attributes of each node can be used to determine thestate of the user interface.

In some embodiments, mapping the one or more interface elements to theone or more simplified user interface elements can include comparing thetree hierarchy or node hierarchy to one or more stored tree hierarchies(or one or more node hierarchies) associated with one or more storedinterface elements. For example, the structure and/or attributes of anode hierarchy associated with the one or more interface elements can becompared to the structure and/or attributes associated with the one ormore stored interface elements (e.g., a one or more interface elementsthat were previously recorded and whose respective structure andattributes were stored as one or more stored interface elements).

Further, mapping the one or more interface elements to the one or moresimplified user interface elements can include determining whether oneor more portions of the tree hierarchy or the node hierarchy match oneor more portions of the one or more stored tree hierarchies or one ormore node hierarchies. For example, similarities and/or differencesbetween the node hierarchy for the one or more interface elements andthe node hierarchy of the one or more stored interface elements can bedetermined based at least in part on the comparison of the treehierarchy of the one or more interface elements to the tree hierarchy ofthe one or more stored interface elements. The one or more portions ofthe one or more user interface elements that match the one or moreportions of the one or more stored interface elements can then be mappedto the one or more simplified user interface elements that areassociated with the one or more stored interface elements.

In some embodiments, the tree hierarchy or node hierarchy can beassociated with an application programming interface (API) that definesone or more attributes of the one or more interface elements. Further,the one or more attributes can include attributes associated with one ormore attributes associated with labels on the one or more interfaceelements (e.g., text labels identifying an interface element), one ormore attributes associated with accessibility markup on the one or moreinterface elements (e.g., accessibility markup associated with enhancingthe ease of use (readability, text size, type of font, and/or contrast)of an interface element), a layout of the one or more interface elements(e.g., the relative position of the one or more interface elementswithin the user interface), attributes associated with one or morerespective inputs of the one or more interface elements, attributesassociated with one or more respective outputs of the one or moreinterface elements, and/or attributes associated with an appearance ofthe one or more interface elements.

In some embodiments, the one or more features can include the pixelsdisplayed on a display device associated with the user interface, theclass name of the current user interface views, the package name of theapplication that is operating in the foreground of the user interface,the accessibility hierarchy associated with the application associatedwith the user interface, and/or the user interface elements displayed onthe user interface, the previous state of the user interface, the timeof day, the currently signed in user, location of the computing deviceassociated with the user interface.

Furthermore, one or more features of the one or more interface elementscan be used in mapping the one or more interface elements to the one ormore simplified user interface elements. The one or more features of theone or more interface elements can include the pixels of an interfaceelement, dimensions and text labels of the interface element; theplacement of an interface element on the user interface; the typeassociated with the interface element; the type of interactions aninterface element can receive; accessibility information including acontent description and geometric position within the node hierarchyassociated with an interface element, a class name of the interfaceelement, a time of day, a location, and/or other user interface specificinformation.

Mapping the one or more interface elements to the one or more simplifieduser interface elements can include using the one or more features, todetermine a distance between the current state of the user interface andone or more recorded states of the user interface that are associatedwith the one or more features. The one or more interface elements can bemapped to the one or more simplified user interface elements based atleast in part on the distance between the current state of the userinterface and one or more recorded states of the user interface that areassociated with the one or more features.

Mapping the one or more interface elements to the one or more simplifieduser interface elements can include determining one or moreprobabilities of each of the one or more interface elements receiving auser interaction. For example, the user interface data can includeinformation associated with the frequency with which one or moreinterface elements received a user interaction (e.g., an interfaceelement receiving input from a user) in the past. Further, the userinterface data can include information associated with the context inwhich a previous user interaction occurred. The frequency with whicheach of the one or more interface elements in a particular context ofthe user interface were interacted with in the past can be used todetermine one or more probabilities of the one or more interfaceelements receiving a user interaction in the present moment.

Further, mapping the one or more interface elements to the one or moresimplified user interface elements can include mapping a predeterminednumber of the one or more simplified user interface elements to arespective number of the one or more interface elements that satisfy oneor more criteria associated with the probability of receiving a userinteraction. The one or more criteria can include a minimum thresholdprobability of an interface element receiving a user interaction and/ora minimum threshold ranking of the one or more interface elements (e.g.,the top three interface elements with the top three highestprobabilities of receiving a user interaction). For example, aninterface element can be mapped to a simplified user interface elementwhen or if the probability of the interface element receiving a userinteraction exceeds a predetermined threshold (e.g., a probability ofninety (90) percent).

Mapping the one or more interface elements to the one or more simplifieduser interface elements can include mapping the one or more interfaceelements to the one or more simplified user interface elements based atleast in part on one or more machine-learned models. The one or moremachine-learned models can be configured and/or trained to determine oneor more features of the one or more interface elements that match theone or more simplified user interface elements. For example, the one ormore machine-learned models can receive input including the one or morefunctionalities of the one or more interface elements and/or the userinterface data. Further, the one or more machine-learned models cangenerate an output including the one or more simplified user interfaceelements that can map to the one or more interface elements.

The mapping of the one or more interface elements to the one or moresimplified user interface elements can include the use of one or morekeys associated with a set of the one or more features of the one ormore interface elements that are determined to correspond to one or morerespective element maps that represent the state of a respective userinterface. Maintaining the consistent appearance of the simplified userinterface elements can include assigning each key in an element map toan element class. The element class can then be assigned to an interfaceelement, and the interface elements belonging to the same class can berepresented using the same interface element. For example, the “CANCEL”interface element that is used to cancel an operation may have adifferent appearance in different applications but may perform thesimilar operation of canceling the operation that was most recentlyinitiated by a user. As such, the “CANCEL” interface elements ofdifferent applications can belong to the same class of interfaceelement. As a result, the features of the “CANCEL” interface elementfrom one application may be useful in generating a simplified userinterface for another application that also has an interface elementthat belongs to the “CANCEL” class of interface elements.

Interface elements can be grouped into element classes by clusteringthem on the basis of the one or more features including the pixels ofthe element's representation; dimensions and text labels on the element;placement on the screen; type of element; type of acceptableinteractions; accessibility information including content descriptionand geometric position within the accessibility node hierarchy; and/orthe class name of the element.

The user computing device can generate a simplified user interface.Further, the simplified user interface can be based at least in part onand/or include any of the one or more simplified user interface elementswhich can include any of the one or more simplified user interfaceelements that are mapped to the one or more interface elements. The oneor more simplified user interface elements can be configured to performone or more operations associated with the one or more functionalities,one or more operations, and/or one or more tasks of the one or moreinterface elements to which the one or more simplified user interfaceelements are mapped. In some embodiments, the simplified user interfacecan include one or more simplified user interface elements that were notmapped to the one or more interface elements (e.g., a simplified userinterface element that initiates operation of an application that isdifferent from the application associated with the user interface).

Generation of the simplified user interface can include classificationof a user interface. Classification of the user interface can includedetermining whether a user interface would be enhanced by a simplifieduser interface. By determining how an application is navigated throughdifferent states of its user interface to complete a task, the one ormore machine-learned models can determine one or more simplified userinterfaces that simplify the user interface through the applicationexperience.

Generation of the simplified user interface can include the predictionof the most probable or likely (e.g., more probable or likely than somethreshold level of probability or likelihood) elements to be interactedwith on the user interface. By predicting probable or likely elements,the one or more interface elements on the screen that would be mostuseful when converted into the one or more simplified user interfaceelements of the simplified user interface.

In some embodiments, the appearance (e.g., the iconography associatedwith a simplified user interface element) of the simplified userinterface can remain consistent across different user interfaces. Forexample, the one or more simplified user interface elements can beconsistent even if the associated interface element corresponding to thesimplified user interface element is presented differently across one ormore different states of a user interface. For example, the “CANCEL”button associated with a simplified user interface element may beconsistently represented in the simplified user interface by the sameicon, though it is represented differently in different applications orin different screens within the same application.

In some embodiments, generating the simplified user interface can bebased at least in part on one more machine-learned models. The one ormore machine-learned models can be configured and/or trained to generatethe simplified user interface based at least in part on an inputincluding the state of the user interface and/or the one or moresimplified user interface elements. For example, based on an inputincluding the state of the user interface and/or a mapping of the userinterface, the one or more machine-learned models can determine alocation, color scheme, and/or orientation of the one or more simplifieduser interface elements within the user interface.

Generation of the simplified user interface can include the use of oneor more machine-learning models to scrape the user interface of anapplication to determine which of the one or more interface elements aretask-directed interface elements and may use natural language output tosummarize these task-directed interface elements into the simplifieduser interface. For example, the simplified user interface for a mappingapplication may generate synthetic voice descriptions of how far eachlocation selected by a user is from the current location of the user.

By way of example, the simplified user interface can include one or moregraphical representations that include a combination of text and imagerywith which a user can interact in order to perform one or moreoperations associated with the one or more functionalities of the one ormore interface elements that are mapped to the one or more simplifieduser interface elements of the simplified user interface.

The user computing device can determine one or more proposed simplifieduser interface elements for each of the one or more interface elementsthat do not match the one or more simplified user interface elements.For example, the user computing device can use one or moremachine-learned models to determine the one or more proposed simplifieduser interface elements based at least in part on an input that includesthe one or more interface elements that do not match the one or moresimplified user interface elements.

The user computing device can generate a prompt that can include arequest for a user to select at least one of the one or more proposedsimplified user interface elements. For example, if the user computingdevice determines that there are three proposed simplified userinterface elements, the user computing device can generate a prompt thatstates “A SIMPLIFIED USER INTERFACE HAS BEEN GENERATED FOR YOU, PLEASESELECT THE USER INTERFACE ELEMENTS THAT YOU ARE LIKELY TO USE IN THEFUTURE.” The user can then manually select (e.g., touch a portion of theuser interface that displays the desired user interface element) theproposed simplified user interface element that the user believes wouldbe useful in the future. The user computing device can also provide anoption for the user to cancel the simplified user interface if the userdetermines that none of the three proposed simplified user interfaceelements are sufficiently valuable to the user.

Generating the simplified user interface can include determining, basedat least in part on the user interface data, one or more locations ofthe one or more interface elements respectively. For example, the usercomputing device can determine the one or more locations within the userinterface of each of the one or more interface elements. The one or morelocations can include a set of coordinates indicating the respectiveposition of each of the one or more interface elements.

Generating the simplified user interface can further include generatingthe simplified user interface based at least in part on the one or morelocations of the one or more interface elements. In some embodiments,the simplified user interface can be generated in a portion of the userinterface in which the fewest of the one or more interface elements arelocated. For example, if the user computing device determines that noneof the one or more interface elements are located on the bottom edge ofa user interface, the simplified user interface can be generated on thebottom edge of the user interface, thereby not obscuring the one or moreinterface elements.

Generating the simplified user interface can include determining, basedat least in part on the user interface data, the one or more interfaceelements that should not be obstructed. For example, the user interfacedata can indicate that an interface element is required to exit or closean application. As such, the interface element that is required to exitor close the application should not be obstructed as doing so wouldprevent a user from exiting or closing the application.

Furthermore, generating the simplified user interface can includegenerating the simplified user interface in one or more portions of theuser interface that do not include the one or more interface elementsthat should not be obstructed. For example, subsequent to determiningthat an interface element (e.g., an exit application interface element)is associated with exiting or closing an application, the user computingdevice can generate the simplified user interface in a portion of theuser interface that does not include the exit application interfaceelement.

In some embodiments, one or more indications can be generated in orderto identify the one or more interface elements that are mapped to theone or more simplified user interface elements. In this way, the one ormore interface elements that have been mapped to the one or moresimplified user interface elements can be readily determined by a user.

The one or more indications can include one or more highlights of theone or more interface elements (e.g., the one or more interface elementsmapped to the one or more simplified user interface elements arehighlighted in a particular color), one or more color changes of the oneor more interface elements (e.g., a particular color or colors areapplied to the one or more interface elements that were mapped to theone or more simplified user interface elements), one or more changes toa brightness of the one or more interface elements (e.g., the one ormore interface elements that are mapped to the one or more simplifieduser interface elements are dimmer or brighter), one or more markings onthe one or more interface elements (e.g., a symbol or text is generatedon or near the one or more interface elements that are mapped to the oneor more simplified user interface elements), and/or one or more shapesaround the one or more interface elements (e.g., a circle or rectangleis generated around the one or more interface elements that are mappedto the one or more simplified user interface elements).

The simplified user interface can be presented (e.g., displayed) in aseparate region from the user interface. For example, the user interfacecan be displayed on one display device and the simplified user interfacecan be displayed on a different display device. By way of furtherexample, the user interface can be displayed in one region of a displaydevice and the simplified user interface can be displayed in a differentregion of the same display device. In some embodiments, the simplifieduser interface can be superimposed over a portion of the user interface.For example, the simplified user interface can be an opaque,transparent, or semi-transparent overlay that is superimposed over aportion of the user interface (e.g., superimposed along the top orbottom edge of a user interface).

The simplified user interface can be associated with control of anapplication. For example, the user interface can be used to receive userinputs that are used to control various operations of an application.Further, the one or more simplified user interface elements can includeone or more predetermined application controls and/or a control toremove the simplified user interface. For example, the one or moresimplified user interface elements can include predetermined applicationcontrols that can be used to control a set of the most recently usedapplications.

The application can include a text messaging application, an e-mailapplication, and/or a telephone application. Further, the one or morepredetermined application controls can include one or more messagetemplates (e.g., text message templates to send predetermined messagesto predetermined users), one or more predetermined geographic locations(e.g., predetermined routes to frequently visited geographic locations),and/or one or more predetermined user contacts (e.g., frequentlycontacted users in a telephone application).

In some embodiments the simplified user interface can be associated withan input modality that is different from that of the user interface. Forexample, the input modality of an interface element of the userinterface can be a visual and tactile input modality in which a userconfirms a route in a mapping application by touching a visual interfaceelement with their finger. The simplified user interface can be an audioinput modality (e.g., voice input modality) in which a user interactswith the simplified user interface for the mapping application by saying“YES” to confirm a route that was generated.

Further, the input modality of the simplified user interface can includea touch input modality (e.g., a visible simplified user interfaceelement that a user can interact with via a touch screen display), anaudio input modality (e.g., a simplified user interface element that auser can interact with via a microphone), and/or a gesture inputmodality (e.g., a simplified user interface element that a user caninteract with via a gesture that is captured by a camera and gesturerecognition system that can capture an image of a gesture and provideinput to the user interface).

In some embodiments the simplified user interface can be associated withan output modality that is different from that of the user interface.For example, the output modality of an interface element of the userinterface can be a visual output modality in which the destination in amapping application is visually represented in a map. The simplifieduser interface can include an audio output modality in which thedestination is announced by the user computing device when the locationis nearby.

In some embodiments, the simplified user interface can be associatedwith, presented on, and/or generated on a different device or systemthan the device or system that the user interface is associated with,presented on, and/or generated on. For example, if the user interface isgenerated on the user computing device (e.g., a laptop computingdevice), the simplified user interface can be associated with, presentedon, and/or or generated on a system or device (e.g., a smartphone) thatis different and/or separate from the user computing device.

By way of further example, the user interface can be presented on adisplay device (e.g., a user interface comprising an instruction toconfirm a user input can be presented via the display device of asmartphone) and the simplified user interface can be presented via oneor more audio output devices (e.g., an auditory instruction to confirm auser input can be presented via headphones and/or earbuds).

Further, the output modality of the simplified user interface caninclude a visual output modality (e.g., a graphical representation ofthe simplified user interface), an auditory output modality (e.g., anauditory description of interface elements of the simplified userinterface), and/or a tactile output modality (e.g., a vibratory outputto indicate the presence of a simplified user interface element).

In some embodiments, the simplified user interface may be used in fullymanual, partly manual, and/or fully automated ways. In the fully-manualembodiment, a user can configure the one or more interface elements thatare communicated via the simplified user interface for one or more oneor more states of a user interface. In the partly-manual embodiment, auser can configure the one or more interface elements with assistancefrom an automated software agent, which can include the use of one ormore machine-learned models. In the fully-automated embodiment, theautomated software agent can be configured to automatically generate thesimplified user interface for one or more states of a user interfacethat are predicted to require and/or benefit from simplification.

The disclosed technology can include a computing system and/or computingdevice (e.g., the user computing device) that is configured to performvarious operations associated with the generation of user interfacesincluding simplified user interfaces. In some embodiments, the usercomputing system can be associated with various computing systems and/ordevices that use, send, receive, and/or generate information and/or dataassociated with the generation of simplified user interfaces.Furthermore, the user computing system can process, generate, modify,and/or access (e.g., send and/or receive) data and/or informationincluding data and/or information associated with user interfaces.

The user computing device can include specialized hardware and/orsoftware that enable the performance of one or more operations specificto the disclosed technology. The user computing system can include oneor more application specific integrated circuits that are configured toperform operations associated with mapping user interface elements tosimplified user interface elements and generating a simplified userinterface.

The systems, methods, devices, apparatuses, and tangible non-transitorycomputer-readable media in the disclosed technology can provide avariety of technical effects and benefits including an improvement inthe generation of user interfaces. In particular, the disclosedtechnology may assist a user (e.g. the user of a user computing device)in performing a technical task by means of a continued and/or guidedhuman-machine interaction process in which a simplified user interfaceis provided to a user, based in part on the state of an existing userinterface. Furthermore, the disclosed technology may also providebenefits including improvements in device ergonomics, better resourceusage efficiency, and improved safety.

The systems, methods, devices, and computer program products (e.g.,non-transitory computer-readable media) in the disclosed technology canprovide a variety of technical effects and benefits to the overallergonomics and ease of use for a user interacting with a user interface.For example, the disclosed technology has the effect that a simplifieduser interface that accepts more ergonomic user interactions can begenerated based on the current configuration of a user interface.

Furthermore, the disclosed technology provides a solution to the problemof excessive user interface complexity by generating a simplified userinterface that reduces the complexity of user interaction in a mannerthat preserves the capabilities of the user interface. Implementationsof the disclosed technology can reduce the number, type, and complexityof burdensome interactions with default user interfaces. This reductionin burdensome interactions (e.g., a user needing to select small userinterface elements or search through buried menus) can, aside fromimproving the ease of use of the user interface, also allow the user toengage the user interface more efficiently, thereby conservingcomputational and battery resources of the user computing device byminimizing the amount of user interaction with the user interface.

Additionally, the disclosed technology can improve user safety byproviding a simplified user interface that may require fewerinteractions by a user. For example, reducing the number of userinteractions with a user interface can result in less diversion of userattention from a task that the user is engaged in. For example, amapping application used by a pedestrian in a busy urban environment mayuse a simplified user interface that allows the user to have more timeto remain aware of their surroundings while interacting with thesimplified user interface of the mapping application.

Further, the disclosed technology can improve safety by generating asimplified user interface that has a different input modality or outputmodality from the initial user interface. For example, in a vehicleenvironment in which diverting the user's attention from the surroundingenvironment can reduce safety, a simplified user interface that acceptsvoice inputs (e.g., voice commands) may be created instead of, or inaddition to, a user interface that requires the user to look at or toucha graphical user interface. As such, the user can interact with thesimplified user interface in a manner that does not require the user todivert their attention from guiding the vehicle.

As such, the disclosed technology may assist the user of a simplifieduser interface to more effectively perform a variety of tasks with thespecific benefits of improved device ergonomics, reduced resourceconsumption, and improved safety. Further, any of the specific benefitsprovided to users can be used to improve the effectiveness of a widevariety of devices and services including any devices or services thatrely on the use of a user interface. Accordingly, the improvementsoffered by the disclosed technology can result in tangible benefits to avariety of applications, devices, and/or systems including mechanical,electronic, and computing systems associated with the generation and useof user interfaces.

With reference now to the Figures, example embodiments of the presentdisclosure will be discussed in further detail.

FIG. 1A depicts a block diagram of an example computing system 100 thatperforms operations associated with the generation of a simplified userinterface according to example embodiments of the present disclosure.The system 100 includes a user computing device 102, a server computingsystem 130, and a training computing system 150 that are communicativelycoupled over a network 180.

The user computing device 102 can be any type of computing device, suchas, for example, a personal computing device (e.g., laptop or desktop),a mobile computing device (e.g., smartphone or tablet), a gaming consoleor controller, a wearable computing device, an embedded computingdevice, or any other type of computing device.

The user computing device 102 includes one or more processors 112 and amemory 114. The one or more processors 112 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 114can include one or more non-transitory computer-readable storagemediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magneticdisks, etc., and combinations thereof. The memory 114 can store data 116and instructions 118 which are executed by the processor 112 to causethe user computing device 102 to perform operations.

In some implementations, the user computing device 102 can store orinclude one or more machine-learned models 120. For example, the one ormore machine-learned models 120 can be or can otherwise include variousmachine-learned models such as neural networks (e.g., deep neuralnetworks) or other types of machine-learned models, including non-linearmodels and/or linear models. Neural networks can include feed-forwardneural networks, recurrent neural networks (e.g., long short-term memoryrecurrent neural networks), convolutional neural networks or other formsof neural networks. Example one or more machine-learned models 120 arediscussed with reference to FIGS. 1A-15.

In some implementations, the one or more machine-learned models 120 canbe received from the server computing system 130 over network 180,stored in the user computing device memory 114, and then used orotherwise implemented by the one or more processors 112. In someimplementations, the user computing device 102 can implement multipleparallel instances of a single machine-learned model 120 (e.g., toperform parallel operations to generate a simplified user interfaceacross multiple instances of a user interface implemented on multiplerespective computing devices).

More particularly, the one or more machine-learned models 120 can beconfigured and/or trained to determine one or more states of a userinterface; map one or more interface elements of the user interface toone or more simplified user interface elements of a simplified userinterface; and/or generate a simplified user interface.

Additionally, or alternatively, one or more machine-learned models 140can be included in or otherwise stored and implemented by the servercomputing system 130 that communicates with the user computing device102 according to a client-server relationship. For example, the one ormore machine-learned models 140 can be implemented by the servercomputing system 140 as a portion of a web service (e.g., a simplifieduser interface generation service). Thus, one or more models 120 can bestored and implemented at the user computing device 102 and/or one ormore models 140 can be stored and implemented at the server computingsystem 130.

The user computing device 102 can also include one or more user inputcomponent 122 that receives user input. For example, the user inputcomponent 122 can be a touch-sensitive component (e.g., atouch-sensitive display screen or a touch pad) that is sensitive to thetouch of a user input object (e.g., a finger or a stylus). Thetouch-sensitive component can serve to implement a virtual keyboard.Other example user input components include a microphone, a traditionalkeyboard, or other means by which a user can provide user input.

The server computing system 130 includes one or more processors 132 anda memory 134. The one or more processors 132 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 134can include one or more non-transitory computer-readable storagemediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magneticdisks, etc., and combinations thereof. The memory 134 can store data 136and instructions 138 which are executed by the processor 132 to causethe server computing system 130 to perform operations.

In some implementations, the server computing system 130 includes or isotherwise implemented by one or more server computing devices. Ininstances in which the server computing system 130 includes pluralserver computing devices, such server computing devices can operateaccording to sequential computing architectures, parallel computingarchitectures, or some combination thereof.

As described above, the server computing system 130 can store orotherwise include one or more machine-learned models 140. For example,the models 140 can be or can otherwise include various machine-learnedmodels. Example machine-learned models include neural networks or othermulti-layer non-linear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neuralnetworks, and convolutional neural networks. Example models 140 arediscussed with reference to FIGS. 1A-15.

The user computing device 102 and/or the server computing system 130 cantrain the models 120 and/or 140 via interaction with the trainingcomputing system 150 that is communicatively coupled over the network180. The training computing system 150 can be separate from the servercomputing system 130 or can be a portion of the server computing system130.

The training computing system 150 includes one or more processors 152and a memory 154. The one or more processors 152 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 154can include one or more non-transitory computer-readable storagemediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magneticdisks, etc., and combinations thereof. The memory 154 can store data 156and instructions 158 which are executed by the processor 152 to causethe training computing system 150 to perform operations. In someimplementations, the training computing system 150 includes or isotherwise implemented by one or more server computing devices.

The training computing system 150 can include a model trainer 160 thattrains the machine-learned models 120 and/or 140 stored at the usercomputing device 102 and/or the server computing system 130 usingvarious training or learning techniques, such as, for example, backwardspropagation of errors. For example, a loss function can bebackpropagated through the model(s) to update one or more parameters ofthe model(s) (e.g., based on a gradient of the loss function). Variousloss functions can be used such as mean squared error, likelihood loss,cross entropy loss, hinge loss, and/or various other loss functions.Gradient descent techniques can be used to iteratively update theparameters over a number of training iterations.

In some implementations, performing backwards propagation of errors caninclude performing truncated backpropagation through time. The modeltrainer 160 can perform a number of generalization techniques (e.g.,weight decays, dropouts, etc.) to improve the generalization capabilityof the models being trained.

In particular, the model trainer 160 can train the one or moremachine-learned models 120 and/or the one or more machine-learned models140 based on a set of training data 162. The training data 162 caninclude, for example, one or more images of user interfaces, one or moreimages of user interface elements, and/or information associated withone or more attributes of the user interfaces or user interfaceelements.

In some implementations, if the user has provided consent, the trainingexamples can be provided by the user computing device 102. Thus, in suchimplementations, the model 120 provided to the user computing device 102can be trained by the training computing system 150 on user-specificdata received from the user computing device 102. In some instances,this process can be referred to as personalizing the model.

The model trainer 160 includes computer logic utilized to providedesired functionality. The model trainer 160 can be implemented inhardware, firmware, and/or software controlling a general purposeprocessor. For example, in some implementations, the model trainer 160includes program files stored on a storage device, loaded into a memoryand executed by one or more processors. In other implementations, themodel trainer 160 includes one or more sets of computer-executableinstructions that are stored in a tangible computer-readable storagemedium such as RAM hard disk or optical or magnetic media.

The network 180 can be any type of communications network, such as alocal area network (e.g., intranet), wide area network (e.g., Internet),or some combination thereof and can include any number of wired orwireless links. In general, communication over the network 180 can becarried via any type of wired and/or wireless connection, using a widevariety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP),encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g.,VPN, secure HTTP, SSL).

The machine-learned models described in this specification may be usedin a variety of tasks, applications, and/or use cases.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be image data. The machine-learned model(s)can process the image data to generate an output. As an example, themachine-learned model(s) can process the image data to generate an imagerecognition output (e.g., a recognition of the image data, a latentembedding of the image data, an encoded representation of the imagedata, a hash of the image data, etc.). As another example, themachine-learned model(s) can process the image data to generate an imagesegmentation output. As another example, the machine-learned model(s)can process the image data to generate an image classification output.As another example, the machine-learned model(s) can process the imagedata to generate an image data modification output (e.g., an alterationof the image data, etc.). As another example, the machine-learnedmodel(s) can process the image data to generate an encoded image dataoutput (e.g., an encoded and/or compressed representation of the imagedata, etc.). As another example, the machine-learned model(s) canprocess the image data to generate an upscaled image data output. Asanother example, the machine-learned model(s) can process the image datato generate a prediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be text or natural language data. Themachine-learned model(s) can process the text or natural language datato generate an output. As an example, the machine-learned model(s) canprocess the natural language data to generate a language encodingoutput. As another example, the machine-learned model(s) can process thetext or natural language data to generate a latent text embeddingoutput. As another example, the machine-learned model(s) can process thetext or natural language data to generate a translation output. Asanother example, the machine-learned model(s) can process the text ornatural language data to generate a classification output. As anotherexample, the machine-learned model(s) can process the text or naturallanguage data to generate a textual segmentation output. As anotherexample, the machine-learned model(s) can process the text or naturallanguage data to generate a semantic intent output. As another example,the machine-learned model(s) can process the text or natural languagedata to generate an upscaled text or natural language output (e.g., textor natural language data that is higher quality than the input text ornatural language, etc.). As another example, the machine-learnedmodel(s) can process the text or natural language data to generate aprediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be speech data. The machine-learned model(s)can process the speech data to generate an output. As an example, themachine-learned model(s) can process the speech data to generate aspeech recognition output. As another example, the machine-learnedmodel(s) can process the speech data to generate a speech translationoutput. As another example, the machine-learned model(s) can process thespeech data to generate a latent embedding output. As another example,the machine-learned model(s) can process the speech data to generate anencoded speech output (e.g., an encoded and/or compressed representationof the speech data, etc.). As another example, the machine-learnedmodel(s) can process the speech data to generate an upscaled speechoutput (e.g., speech data that is of higher quality than the inputspeech data, etc.). As another example, the machine-learned model(s) canprocess the speech data to generate a textual representation output(e.g., a textual representation of the input speech data, etc.). Asanother example, the machine-learned model(s) can process the speechdata to generate a prediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be latent encoding data (e.g., a latent spacerepresentation of an input, etc.). The machine-learned model(s) canprocess the latent encoding data to generate an output. As an example,the machine-learned model(s) can process the latent encoding data togenerate a recognition output. As another example, the machine-learnedmodel(s) can process the latent encoding data to generate areconstruction output. As another example, the machine-learned model(s)can process the latent encoding data to generate a search output. Asanother example, the machine-learned model(s) can process the latentencoding data to generate a reclustering output. As another example, themachine-learned model(s) can process the latent encoding data togenerate a prediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be statistical data. The machine-learnedmodel(s) can process the statistical data to generate an output. As anexample, the machine-learned model(s) can process the statistical datato generate a recognition output. As another example, themachine-learned model(s) can process the statistical data to generate aprediction output. As another example, the machine-learned model(s) canprocess the statistical data to generate a classification output. Asanother example, the machine-learned model(s) can process thestatistical data to generate a segmentation output. As another example,the machine-learned model(s) can process the statistical data togenerate a segmentation output. As another example, the machine-learnedmodel(s) can process the statistical data to generate a visualizationoutput. As another example, the machine-learned model(s) can process thestatistical data to generate a diagnostic output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be sensor data. The machine-learned model(s)can process the sensor data to generate an output. As an example, themachine-learned model(s) can process the sensor data to generate arecognition output. As another example, the machine-learned model(s) canprocess the sensor data to generate a prediction output. As anotherexample, the machine-learned model(s) can process the sensor data togenerate a classification output. As another example, themachine-learned model(s) can process the sensor data to generate asegmentation output. As another example, the machine-learned model(s)can process the sensor data to generate a segmentation output. Asanother example, the machine-learned model(s) can process the sensordata to generate a visualization output. As another example, themachine-learned model(s) can process the sensor data to generate adiagnostic output. As another example, the machine-learned model(s) canprocess the sensor data to generate a detection output.

In some cases, the machine-learned model(s) can be configured to performa task that includes encoding input data for reliable and/or efficienttransmission or storage (and/or corresponding decoding). For example,the task may be an audio compression task. The input may include audiodata and the output may comprise compressed audio data. In anotherexample, the input includes visual data (e.g. one or more images orvideos), the output comprises compressed visual data, and the task is avisual data compression task. In another example, the task may comprisegenerating an embedding for input data (e.g. input audio or visualdata).

In some cases, the input includes visual data and the task is a computervision task. In some cases, the input includes pixel data for one ormore images and the task is an image processing task. For example, theimage processing task can be image classification, where the output is aset of scores, each score corresponding to a different object class andrepresenting the likelihood that the one or more images depict an objectbelonging to the object class. The image processing task may be objectdetection, where the image processing output identifies one or moreregions in the one or more images and, for each region, a likelihoodthat region depicts an object of interest. As another example, the imageprocessing task can be image segmentation, where the image processingoutput defines, for each pixel in the one or more images, a respectivelikelihood for each category in a predetermined set of categories. Forexample, the set of categories can be foreground and background. Asanother example, the set of categories can be object classes. As anotherexample, the image processing task can be depth estimation, where theimage processing output defines, for each pixel in the one or moreimages, a respective depth value. As another example, the imageprocessing task can be motion estimation, where the network inputincludes multiple images, and the image processing output defines, foreach pixel of one of the input images, a motion of the scene depicted atthe pixel between the images in the network input.

In some cases, the input includes audio data representing a spokenutterance and the task is a speech recognition task. The output maycomprise a text output which is mapped to the spoken utterance. In somecases, the task comprises encrypting or decrypting input data. In somecases, the task comprises a microprocessor performance task, such asbranch prediction or memory address translation.

FIG. 1A illustrates one example computing system that can be used toimplement the present disclosure. Other computing systems can be used aswell. For example, in some implementations, the user computing device102 can include the model trainer 160 and the training dataset 162. Insuch implementations, the models 120 can be both trained and usedlocally at the user computing device 102. In some of suchimplementations, the user computing device 102 can implement the modeltrainer 160 to personalize the models 120 based on user-specific data.

FIG. 1B depicts a block diagram of an example computing device 10 thatperforms according to example embodiments of the present disclosure. Thecomputing device 10 can be a user computing device or a server computingdevice.

The computing device 10 includes a number of applications (e.g.,applications 1 through N). Each application contains its own machinelearning library and machine-learned model(s). For example, eachapplication can include a machine-learned model. Example applicationsinclude a text messaging application, an email application, a dictationapplication, a virtual keyboard application, a browser application, etc.

As illustrated in FIG. 1B, each application can communicate with anumber of other components of the computing device, such as, forexample, one or more sensors, a context manager, a device statecomponent, and/or additional components. In some implementations, eachapplication can communicate with each device component using an API(e.g., a public API). In some implementations, the API used by eachapplication is specific to that application.

FIG. 1C depicts a block diagram of an example computing device 50 thatperforms according to example embodiments of the present disclosure. Thecomputing device 50 can be a user computing device or a server computingdevice.

The computing device 50 includes a number of applications (e.g.,applications 1 through N). Each application is in communication with acentral intelligence layer. Example applications include a textmessaging application, an email application, a dictation application, avirtual keyboard application, a browser application, etc. In someimplementations, each application can communicate with the centralintelligence layer (and model(s) stored therein) using an API (e.g., acommon API across all applications).

The central intelligence layer includes a number of machine-learnedmodels. For example, as illustrated in FIG. 1C, a respectivemachine-learned model (e.g., a model) can be provided for eachapplication and managed by the central intelligence layer. In otherimplementations, two or more applications can share a singlemachine-learned model. For example, in some implementations, the centralintelligence layer can provide a single model (e.g., a single model) forall of the applications. In some implementations, the centralintelligence layer is included within or otherwise implemented by anoperating system of the computing device 50.

The central intelligence layer can communicate with a central devicedata layer. The central device data layer can be a centralizedrepository of data for the computing device 50. As illustrated in FIG.1C, the central device data layer can communicate with a number of othercomponents of the computing device, such as, for example, one or moresensors, a context manager, a device state component, and/or additionalcomponents. In some implementations, the central device data layer cancommunicate with each device component using an API (e.g., a privateAPI).

FIG. 2 depicts a block diagram of an example of one or moremachine-learned models 200 according to example embodiments of thepresent disclosure. In some implementations, the one or moremachine-learned models 200 are trained to receive a set of input data204 descriptive of a user interface (e.g., the state of a user interfaceassociated with the user interface data) and, after performing one ormore operations on the input data 204, generating output data 206 thatincludes a simplified user interface. Thus, in some implementations, theone or more machine-learned models 200 can include a simplified userinterface machine-learned model 202 that is operable to generate outputassociated with a simplified user interface.

FIG. 3 depicts a diagram of an example user computing device accordingto example embodiments of the present disclosure. A user computingdevice 300 can include one or more attributes and/or capabilities of thecomputing device 102, the computing system 130, and/or the trainingcomputing system 150. Furthermore, the user computing device 300 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, and/or the training computing system 150, whichare depicted in FIG. 1A.

As shown in FIG. 3, the user computing device 300 can include one ormore memory devices 302, user interface data 304, simplified userinterface data 306, one or more machine-learned models 308, one or moreinterconnects 310, one or more processors 320, a network interface 322,one or more mass storage devices 324, one or more output devices 326,one or more sensors 328, one or more input devices 330, and/or thelocation device 332.

The one or more memory devices 302 can store information and/or data(e.g., the user interface data 304, the simplified user interface data306, and/or the one or more machine-learned models 308). Further, theone or more memory devices 302 can include one or more non-transitorycomputer-readable storage media, including RAM, ROM, EEPROM, EPROM,flash memory devices, magnetic disks, and combinations thereof. Theinformation and/or data stored by the one or more memory devices 302 canbe executed by the one or more processors 320 to cause the usercomputing device 300 to perform operations including operationsassociated with generating a user interface and/or a simplified userinterface.

The user interface data 304 can include one or more portions of data(e.g., the data 116, the data 136, and/or the data 156, which aredepicted in FIG. 1A) and/or instructions (e.g., the instructions 118,the instructions 138, and/or the instructions 158 which are depicted inFIG. 1A) that are stored in the memory 114, the memory 134, and/or thememory 154 respectively. Furthermore, the user interface data 304 caninclude information associated with one or more aspects of a userinterface (e.g., one or more attributes of one or more interfaceelements) that can be implemented on the user computing device 300. Insome embodiments, the user interface data 304 can be received from oneor more computing systems (e.g., the computing system 130 that isdepicted in FIG. 1) which can include one or more computing systems thatare remote (e.g., in another room, building, part of town, city, ornation) from the computing device 300.

The simplified user interface data 306 can include one or more portionsof data (e.g., the data 116, the data 136, and/or the data 156, whichare depicted in FIG. 1A) and/or instructions (e.g., the instructions118, the instructions 138, and/or the instructions 158 which aredepicted in FIG. 1A) that are stored in the memory 114, the memory 134,and/or the memory 154 respectively. Furthermore, the user interface data306 can include information associated with one or more aspects of asimplified user interface (e.g., one or more attributes of one or moresimplified user interface elements) that can be implemented on the usercomputing device 300. In some embodiments, the simplified user interfacedata 306 can be received from one or more computing systems (e.g., thecomputing system 130 that is depicted in FIG. 1) which can include oneor more computing systems that are remote from the computing device 300.

The one or more machine-learned models 308 (e.g., the one or moremachine-learned models 120 and/or the one or more machine-learned models140) can include one or more portions of the data 116, the data 136,and/or the data 156 which are depicted in FIG. 1A and/or instructions(e.g., the instructions 118, the instructions 138, and/or theinstructions 158 which are depicted in FIG. 1A) that are stored in thememory 114, the memory 134, and/or the memory 154 respectively.Furthermore, the one or more machine-learned models 308 can includeinformation associated with determining the state of a user interface,mapping one or more interface elements to one or more simplified userinterface elements, and/or generating a simplified user interface. Insome embodiments, the one or more machine-learned models 308 can bereceived from one or more computing systems (e.g., the computing system130 that is depicted in FIG. 1) which can include one or more computingsystems that are remote from the computing device 300.

The one or more interconnects 310 can include one or more interconnectsor buses that can be used to send and/or receive one or more signals(e.g., electronic signals) and/or data (e.g., the user interface data304, the simplified user interface data 306, and/or the one or moremachine-learned models 308) between components of the user computingdevice 300, including the one or more memory devices 302, the one ormore processors 320, the network interface 322, the one or more massstorage devices 324, the one or more output devices 326, the one or moresensors 328 (e.g., a sensor array), and/or the one or more input devices330. The one or more interconnects 310 can be arranged or configured indifferent ways including as parallel or serial connections. Further theone or more interconnects 310 can include one or more internal buses toconnect the internal components of the user computing device 300; andone or more external buses used to connect the internal components ofthe user computing device 300 to one or more external devices. By way ofexample, the one or more interconnects 310 can include differentinterfaces including Industry Standard Architecture (ISA), Extended ISA,Peripheral Components Interconnect (PCI), PCI Express, Serial ATAttachment (SATA), HyperTransport (HT), USB (Universal Serial Bus),Thunderbolt®, IEEE 1394 interface (FireWire®), and/or other interfacesthat can be used to connect components.

The one or more processors 320 can include one or more computerprocessors that are configured to execute the one or more instructionsstored in the one or more memory devices 302. For example, the one ormore processors 320 can, for example, include one or more generalpurpose central processing units (CPUs), application specific integratedcircuits (ASICs), and/or one or more graphics processing units (GPUs).Further, the one or more processors 320 can perform one or more actionsand/or operations including one or more actions and/or operationsassociated with the user interface data 304, the simplified userinterface data 306, and/or the one or more machine-learned models 308.The one or more processors 320 can include single or multiple coredevices including a microprocessor, microcontroller, integrated circuit,and/or a logic device.

The network interface 322 can support network communications. Forexample, the network interface 322 can support communication vianetworks including a local area network and/or a wide area network(e.g., the Internet). The one or more mass storage devices 324 (e.g., ahard disk drive and/or a solid state drive) can be used to store dataincluding the location data 304, the image data 306, and/or theorientation data 308. The one or more output devices 326 can include oneor more display devices (e.g., LCD display, OLED display, Mini-LEDdisplay, microLED display, plasma display, and/or CRT display), one ormore light sources (e.g., LEDs), one or more loudspeakers, and/or one ormore haptic output devices (e.g., one or more devices that areconfigured to generate vibratory output).

The one or more input devices 330 can include one or more keyboards, oneor more touch sensitive devices (e.g., a touch screen display), one ormore buttons (e.g., ON/OFF buttons and/or YES/NO buttons), one or moremicrophones, and/or one or more cameras.

The one or more memory devices 302 and the one or more mass storagedevices 324 are illustrated separately, however, the one or more memorydevices 302 and the one or more mass storage devices 324 can be regionswithin the same memory module. The user computing device 300 can includeone or more additional processors, memory devices, network interfaces,which may be provided separately or on the same chip or board. The oneor more memory devices 302 and the one or more mass storage devices 324can include one or more computer-readable media, including, but notlimited to, non-transitory computer-readable media, RAM, ROM, harddrives, flash drives, and/or other memory devices.

The one or more memory devices 302 can store sets of instructions forapplications including an operating system that can be associated withvarious software applications or data. For example, the one or morememory devices 302 can store sets of instructions for applications thatinclude a user interface that can receive input and generate output. Theone or more memory devices 302 can be used to operate variousapplications including a mobile operating system developed specificallyfor mobile devices. As such, the one or more memory devices 302 canstore instructions that allow the software applications to access dataincluding data associated with the generation of a user interface and/ora simplified user interface. In other embodiments, the one or morememory devices 302 can be used to operate or execute a general-purposeoperating system that operates on both mobile and stationary devices,including for example, smartphones, laptop computing devices, tabletcomputing devices, and/or desktop computers.

The software applications that can be operated or executed by the usercomputing device 300 can include applications associated with the system100 shown in FIG. 1A. Further, the software applications that can beoperated and/or executed by the user computing device 300 can includenative applications and/or web-based applications.

The location device 332 can include one or more devices or circuitry fordetermining the position of the user computing device 300. For example,the location device 332 can determine an actual and/or relative positionof the user computing device 300 by using a satellite navigationpositioning system (e.g. a GPS system, a satellite positioning systemwith trade name Galileo, a satellite positioning system with trade nameGLObal Navigation satellite system (GLONASS), a satellite positioningsystem with trade name BeiDou Satellite Navigation and Positioningsystem), an inertial navigation system, a dead reckoning system, basedon IP address, by using triangulation and/or proximity to cellulartowers or Wi-Fi hotspots, beacons, and the like and/or other suitabletechniques for determining position.

FIG. 4 depicts an example of interactions between a user and a userinterface implemented on a user computing device according to exampleembodiments of the present disclosure. A user computing device 402 caninclude one or more attributes and/or capabilities of the computingdevice 102, the computing system 130, the training computing system 150,and/or the user computing device 300. Furthermore, the user computingdevice 402 can perform one or more actions and/or operations includingthe one or more actions and/or operations performed by the computingdevice 102, the computing system 130, and/or the training computingsystem 150, and/or the user computing device 300.

As shown in FIG. 4, the environment 400 includes the user computingdevice 402, a user interface 404, one or more task-directed outputs 406,one or more task-auxiliary outputs 408, one or more user interactions410, one or more task-directed inputs 412, and one or moretask-auxiliary inputs 414.

A user can use a computing device (e.g., the user computing device 300depicted in FIG. 3) to operate and/or control various applications. Forexample, a user can use a smartphone that can run an application,receive one or more inputs from the user, and generate one or moreoutputs that can be consumed by the user. The one or more inputs and/orone or more outputs can include one or more task-directed inputs 412 tothe user computing device 402 and/or one or more task-directed outputs406 from the user computing device 402, that may be associated with theuser interactions 410 with the user interface 404 that directly resultin the performance and/or completion of a task. The one or more inputsand/or one or more outputs can also include one or more task-auxiliaryinputs 414 to the user computing device 402 and/or one or moretask-auxiliary outputs 408 from the user computing device 402. The oneor more task-auxiliary inputs 412 and/or one or more task-auxiliaryoutputs 408 can be associated with the user interactions 410 with theuser interface 404 that may not necessarily be directly related to theperformance and/or completion of a task. Further, the one or moretask-auxiliary inputs 414 and/or the one or more task-auxiliary outputs408 may assist a user in communicating and/or interacting with the usercomputing device 402 through the user interface 404.

For example, in a ride-hailing application that is implemented on theuser computing device 402, the one or more task-directed inputs 412 caninclude the current location of the user computing device 402 anddestination of a user associated with the user computing device 402 410(e.g., a location to which the user of the user computing device 402plans to travel), and the type of vehicle (e.g., a sedan, a compact car,and/or a van) that the user of the user computing device 402 requested.The one or more task-directed outputs 406 can include a confirmationmessage indicating that the requested vehicle is on its way, anestimated time of arrival of the vehicle (“ETA”), and the payment thatwill be required to pay the bill for the requested vehicle.

In some embodiments, in order to communicate the one or moretask-directed inputs 412 to the user computing device 402, the userinterface 404 may include one or more interface elements (not shown).For example, the user interface 404 can include a map interface elementthat can be used to assist a user in the selection of a source locationand a destination location for a trip; one or more vehicle selectioninterface elements to assist the user in the selection of differenttypes of vehicles; and/or various other interface elements that a usercan use to confirm their selections.

In another example, one or more task-directed inputs 412 can include auser making a voice input (e.g., speaking to another user at the otherend of the telephone connection); and task-directed outputs 406including a user receiving voice output from the user at the other endof the telephone connection. Further, the one or more task-auxiliaryinputs 414 can include a user controlling one or more interface elements(e.g., interface elements visually represented as buttons on the userinterface 404) that allow the user to exit a telephone call, switch tospeaker mode, and/or display the numeric keypad.

In some embodiments, one or more task-directed inputs 412 and/or the oneor more task-directed outputs 406 can be transformed and/or simplified.The transformation and/or simplification of the one or moretask-directed inputs 412 and/or the one or more task-directed outputs406 can be conditional on no information being lost in thetransformation or simplification. For example, when making a telephonecall accessible to people who are hearing impaired, a simplificationalgorithm may caption the telephone call (since captions can be 1:1mappable back to voice, thereby not losing textual information in thetransformation). In contrast, the performance of other types oftransformations and/or simplifications, for example, the summarizationand/or simplification of language, may result in a task-level change inthe application which may not represent the user's intention accurately.

Furthermore, the one or more task-auxiliary inputs 414 and/or the one ormore task-auxiliary outputs 408 may be related to the modalities (e.g.,input modalities and/or output modalities) that are available on theuser computing device 402 and that encode implicit assumptions about theuser's capabilities and the device's capabilities. In some embodiments,the one or more task-auxiliary inputs 414 and/or the one or moretask-auxiliary outputs 408 can be changed in significant ways if needed,without affecting the task that is being implemented on the application.For example, if the user computing device 402 is a smartphone thatrelies on a display screen as the primary output and touch as theprimary input, the user computing device 402 may offer a user interfacethat primarily relies on buttons and edit text boxes for input, withlabels that guide the user to use these input widgets. A simplificationalgorithm may be able to change the modality of the one or moretask-auxiliary inputs 414 and the one or more task-auxiliary outputs408. For example, in a telephone communication application, asimplification algorithm may hide several less-important interfaceelements and only show the one most relevant and/or frequently usedinterface elements (e.g. the hang-up call interface element).

FIG. 5 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 500 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, and/or thecomputing device 300. Furthermore, the user computing device 500 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, the training computing system 150, and/or thecomputing device 300.

As shown in FIG. 5, the user computing device 500 includes a displaycomponent 502, an imaging component 504, an audio input component 506, atactile component 508, a tactile component 510, an audio outputcomponent 512, a user interface 514, a simplified user interface 516, aninterface element 518, an interface element 520, a simplified userinterface element 522, and a simplified user interface element 524.

The user computing device 500 can be configured to receive one or moreinputs including data (e.g., application data associated with aride-share application implemented on the user computing device 500and/or user interface data) and/or one or more user interactions from auser of the computing device 500). Further, the user computing device500 can be configured to generate the simplified user interface 516 thatsimplifies one or more inputs and/or one or more outputs of the userinterface 514. The user interface 514 and/or the simplified userinterface 516 can receive inputs and/or outputs via any of the displaycomponent 502, the imaging component 504, the audio input component 506,the tactile component 508, and/or the tactile component 510. Further,the user interface 514 and/or the simplified user interface 516 cangenerate outputs using any of the display component 502, the tactilecomponent 508, the tactile component 510, and/or the audio outputcomponent 512.

The display component 502 can be configured to receive one or moreinputs and/or generate one or more outputs associated with the userinterface 514 and/or the simplified user interface 516. For example, thedisplay component 502 can be configured to receive inputs to the userinterface 514 including touch inputs associated with the selection of avehicle for the user of the user computing device 500. Further, thedisplay component 502 can be configured to generate one or more outputsincluding the simplified user interface 516, which can include asimplified version of the one or more interface elements of the userinterface 514.

The imaging component 504 can include one or more cameras that can beconfigured to receive input including one or more images. The one ormore images received by the imaging component 504 can be used in thegeneration of a simplified user interface 516 by receiving visualinformation (e.g., an image of a user gesture) that can be used as analternative to another input modality (e.g., a user touching the displaycomponent 502). For example, a mapping application implemented on theuser interface 514 can receive touch inputs to the display component 502as the default input to confirm the selection of a vehicle. Thesimplified user interface 516 can use the imaging component 504 toreceive visual inputs to indicate confirmation of the selection of avehicle (e.g., an image of the user making the thumbs up gesture toconfirm a selection).

The audio input component 506 can include one or more microphones thatcan be configured to receive sound inputs including speech from a userof the user computing device 500. The simplified user interface 516generated on the user computing device 500 can include a simplificationof visual interface elements (e.g., a text prompt to select a vehicle)by using the audio input component to receive auditory inputs toindicate confirmation of the selection of a vehicle (e.g., the usersaying the words “vehicle confirmed” or just “confirmed”).

The tactile components 508/510 can receive inputs including tactileinputs including touches and presses, and can in some embodiments,generate outputs including haptic feedback (e.g., vibratory feedback).The simplified user interface 516 generated on the user computing device500 can include a simplification of visual interface elements (e.g.,zooming into the map) by using the tactile components 508/510 to receivetactile inputs (e.g., pressing the tactile component 508) to indicatezooming into the map.

Furthermore, the user computing device 500 can be configured to generateoutputs (e.g., visual outputs including the user interface 514 and/orthe simplified user interface 516) that include the locations ofvehicles that a user can select. The user computing device 500 cansimplify the user interface 514 by generating the simplified userinterface 516 that includes one or more outputs that have a differentoutput modality from the output modality provided by the user interface514. For example, the user interface 514 can include textualnotifications when a vehicle selected by a user is ready to pick up theuser. The simplified user interface 516 can use the audio outputcomponent 512 (e.g., one or more loudspeakers) to generate audibleoutputs including audible notifications (e.g., an announcement that “thevehicle is ready to pick you up”) when a vehicle selected by a user isready to pick up the user.

In this example, the user computing device 500 is implementing aride-sharing application that displays a map of the geographic areawithin a two (2) kilometer radius around the user computing device 500.The user interface 514 generates the interface element 526 (e.g., a mapinterface element that renders an image of a geographic area); theinterface element 518 which represents a first vehicle (“VEHICLE 1”)that a user can select by touching (e.g., tapping) the interface element518; and the interface element 520 which represents a second vehicle(“VEHICLE 2”) that a user can select by touching the interface element520. The user computing device 500 can determine the state of the userinterface 514 (e.g., that the user interface 514 is associated with aride-share application including the one or more interface elements518/520) and can generate the simplified user interface 516.

The simplified user interface 516 can include the simplified userinterface element 522 (“CONFIRM”) that a user can use to confirm aselection within the user interface 514; and the simplified userinterface element 524 (“CANCEL”) that a user can use to cancel aselection within the user interface 514. For example, the applicationmay default to the interface element 518 (“VEHICLE 1”) and if the userwould like to confirm selection of the interface element 518, the usercan simply touch the simplified user interface element 522 to confirmtheir selection instead of potentially incorrectly tapping interfaceelement 520 or possibly having to interact with an additional interfaceelement to confirm their selection.

FIG. 6 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 600 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300. Furthermore, the user computing device 600 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300.

As shown in FIG. 6, the user computing device 600 includes a displaycomponent 602, an imaging component 604, an audio input component 606, atactile component 608, a tactile component 610, an audio outputcomponent 612, a user interface 614, a simplified user interface 616, aninterface element 618, an interface element 620, a simplified userinterface element 622, a simplified user interface element 624, asimplified user interface element 626, and a simplified user interfaceelement 628.

The user computing device 600 can be configured to receive one or moreinputs including data (e.g., application data associated withcommunication application (a telephone and video-call application)implemented on the user computing device 600 and/or user interface data)and/or one or more user interactions from a user of the computing device600). Further, the user computing device 600 can be configured togenerate the simplified user interface 616 that simplifies one or moreinputs and/or one or more outputs of the user interface 614. The userinterface 614 and/or the simplified user interface 616 can receiveinputs and/or outputs via any of the display component 602, the imagingcomponent 604, the audio input component 606, the tactile component 608,and/or the tactile component 610. Further, the user interface 614 and/orthe simplified user interface 616 can generate outputs using any of thedisplay component 602, the tactile component 608, the tactile component610, and/or the audio output component 612.

The display component 602 can be configured to receive one or moreinputs and/or generate one or more outputs associated with the userinterface 614 and/or the simplified user interface 616. For example, thedisplay component 602 can be configured to receive inputs to the userinterface 614 including touch inputs associated with the dialing atelephone number, calling a predetermined telephone number, and/orinitiating a call. Further, the display component 602 can be configuredto generate one or more outputs including the simplified user interface616, which can include a simplified version of the one or more interfaceelements of the user interface 614.

The imaging component 604 can include one or more cameras that can beconfigured to receive input including one or more images. The one ormore images received by the imaging component 604 can be used in thegeneration of the simplified user interface 616 by receiving visualinformation (e.g., an image of a user) that can be used as analternative to another input modality (e.g., a user entering a passcodeto access the communication application). For example, a communicationapplication implemented on the user interface 614 can receive touchinputs to the display component 602 as the default input to enter apasscode to access the communication application and make calls. Thesimplified user interface 616 can use the imaging component 604 toreceive visual inputs to confirm the identity of a user (e.g., an imageof the user's face) through use of a facial recognition applicationimplemented on the user computing device 600.

The audio input component 606 can include one or more microphones thatcan be configured to receive sound inputs including speech from a userof the user computing device 600. The simplified user interface 616generated on the user computing device 600 can include a simplificationof visual interface elements (e.g., the numeric keypad interfaceelement) by using the audio input component to receive auditory inputsto that indicate a telephone number or contact that a user is attemptingto call (e.g., the user speaking a telephone number or saying “callhome”).

The tactile components 608/610 can receive inputs including tactileinputs including touches and presses. The simplified user interface 616generated on the user computing device 600 can include a simplificationof visual interface elements (e.g., hanging up on a call) by using thetactile components 608/610 to receive tactile inputs (e.g., pressing thetactile component 608) to hang up on a call.

Furthermore, the user computing device 600 can be configured to generateoutputs that include one or more auditory outputs associated with callsreceived by the communication application. The user computing device 600can simplify the user interface 614 by generating the simplified userinterface 616 that includes one or more outputs that have a differentoutput modality from the output modality provided by the user interface614. For example, the user interface 614 can include textualnotifications a call was missed. The simplified user interface 616 canuse the audio output component 612 (e.g., one or more loudspeakers) togenerate audible outputs including audible notifications (e.g., “youmissed a call from the office”) when a vehicle associated with themissed call.

In this example, the user computing device 600 is implementing acommunication application (e.g., a telephone application that can beused to make voice calls and/or video calls). The user interface 614generates a numeric keypad, the interface element 618 which can be usedto initiate a call; and the interface element 620 which can be used toend a call. The interface element 618 and the interface element 620 canbe selected by a user touching (e.g., pressing) the interface element618 and/or the interface element 620 respectively.

The user computing device 600 can use user interface data associatedwith the state of the user interface 614 to determine the state of theuser interface 614. For example, the user computing device 600 candetermine that the user interface 614 is associated with a communicationapplication and can further determine one or more locations and/or oneor more functionalities of the interface elements of the user interface614 including the interface element 618 and the interface element 620.

Further, based on the state of the user interface 614 the user computingdevice 600 can generate the simplified user interface 616. Thesimplified user interface can be based at least in part on a user'sprevious interactions (e.g., calls made and calls received) with theuser interface 614, which can be stored in user interface data and whichcan be used to configure and/or train one or more machine-learned modelsthat can be used to determine the state of the user interface 614 and/orgenerate the simplified user interface 616. For example, the usercomputing device 600 can use the user's past interactions to determinethe most frequently called telephone numbers and can then generatesimplified user interface elements that are associated with thosetelephone numbers in order to facilitate a user's access to thosetelephone numbers.

The simplified user interface 616 can include the simplified userinterface element 622 (“CONFIRM”) that a user can use to confirm aselection within the user interface 614. For example, the simplifieduser interface element 622 can be used to confirm the addition of athird caller to a voice call (e.g., a three-way conference call).Further, the simplified user interface 616 can include the simplifieduser interface element 624 (“CANCEL”) that a user can use to undo orcancel an operation and/or selection within the user interface 614. Forexample, a user can use the simplified user interface element 624 toundo a number that was incorrectly selected on the numeric keypad.

Furthermore, the simplified user interface 616 can include thesimplified user interface element 626 (“HOME”) that a user can use tomake a call to a predetermined telephone number or personal contact thatis associated with the simplified user interface element 626. Forexample, a user can use the simplified user interface element to callthe home telephone number of their close relative. The simplified userinterface 616 can further include the simplified user interface element628 (“OFFICE”) that a user can use to make a call to a predeterminedtelephone number or personal contact that is associated with thesimplified user interface element 626. For example, a user can use thesimplified user interface element 628 to call their office or businesstelephone number.

FIG. 7 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 700 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300. Furthermore, the user computing device 700 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300.

As shown in FIG. 7, the user computing device 700 includes a displaycomponent 702, an imaging component 704, an audio input component 706, atactile component 708, a tactile component 710, an audio outputcomponent 712, a user interface 714, a simplified user interface 716, aninterface element 718, an interface element 720, a simplified userinterface element 722, a simplified user interface element 724, asimplified user interface element 726, and a simplified user interfaceelement 728.

The user computing device 700 can be configured to receive one or moreinputs including data (e.g., application data associated with amessaging application implemented on the user computing device 700and/or user interface data) and/or one or more user interactions from auser of the computing device 700). Further, the user computing device700 can be configured to generate the simplified user interface 716 thatsimplifies one or more inputs and/or one or more outputs of the userinterface 714. The user interface 714 and/or the simplified userinterface 716 can receive inputs and/or outputs via any of the displaycomponent 702, the imaging component 704, the audio input component 706,the tactile component 708, and/or the tactile component 710. Further,the user interface 714 and/or the simplified user interface 716 cangenerate outputs using any of the display component 702, the tactilecomponent 708, the tactile component 710, and/or the audio outputcomponent 712.

The display component 702 can be configured to receive one or moreinputs and/or generate one or more outputs associated with the userinterface 714 and/or the simplified user interface 716. For example, thedisplay component 702 can be configured to receive inputs to the userinterface 714 including touch inputs associated with initiating a textmessage chat, sending a text message, and/or accessing a previouslyreceived text message. Further, the display component 702 can beconfigured to generate one or more outputs including the simplified userinterface 716, which can include a simplified version of the one or moreinterface elements of the user interface 714.

The imaging component 704 can include one or more cameras that can beconfigured to receive input including one or more images. The one ormore images received by the imaging component 704 can be used in thegeneration of the simplified user interface 716 by receiving visualinformation (e.g., an image of a user's eyes) that can be used as analternative to another input modality (e.g., a user touching the displaycomponent 702). For example, a text messaging application implemented onthe user interface 714 can receive touch inputs to the display component702 as the default input to type “YES” or “NO” in response to aquestion. The simplified user interface 716 can use the imagingcomponent 704 to receive visual inputs of a user's head and can use oneor more image recognition techniques to determine that a head noddinggesture indicates “YES” and a head turning from one side to the otherside and back indicates “NO” in response to a question.

The audio input component 706 can include one or more microphones thatcan be configured to receive sound inputs including speech from a userof the user computing device 700. The simplified user interface 716generated on the user computing device 700 can include a simplificationof visual interface elements (e.g., a pop-up on-screen keyboard that canbe used to enter text) by using the audio input component to receiveauditory inputs (e.g., voice dictation) of a text message that a usercomposes by speaking the text message aloud.

The tactile components 708/710 can receive inputs including tactileinputs including touches and presses. The simplified user interface 716generated on the user computing device 700 can include a simplificationof visual interface elements (e.g., touching the interface element 720to send a text message) by using the tactile components 708/710 toreceive tactile inputs (e.g., pressing the tactile component 708) tosend a text message.

Furthermore, the user computing device 700 can be configured to generateoutputs (e.g., visual outputs including the user interface 714 and/orthe simplified user interface 716) that are associated with the textmessage application. The user computing device 700 can simplify the userinterface 714 by generating the simplified user interface 716 thatincludes one or more outputs that have a different output modality fromthe output modality provided by the user interface 714. For example, theuser interface 714 can generate visual outputs that display the contentsof a text message on the display component 702. The simplified userinterface 716 can use the audio output component 712 (e.g., one or moreloudspeakers) to generate audible outputs that use a synthetic voice toread aloud the contents of a text message.

In this example, the user computing device 700 is implementing amessaging application (e.g., a text messaging application that can beused to communicate with one or more users). The user interface 714includes the interface element 718 (“START CHAT”) which can be used toinitiate a text message; and the interface element 720 (“SEND”) whichcan be used to send a text message. The interface element 718 and/or theinterface element 720 can be respectively selected by a user touching(e.g., pressing) the interface element 718 or the interface element 720.

The user computing device 700 can determine the state of the userinterface 714 and can generate the simplified user interface 716.Further, the user computing device 700 can use user interface dataassociated with the state of the user interface 714 to determine thestate of the user interface 714. For example, the user computing device700 can determine that the user interface 714 is associated with amessaging application and can further determine one or more locationsand/or one or more functionalities (e.g., initiating a message and/orsending a message) of the interface elements of the user interface 714including the interface element 718 and the interface element 720.

Further, based on the state of the user interface 714 the user computingdevice 700 can generate the simplified user interface 716. Thesimplified user interface can be based at least in part on a user'sprevious interactions with the user interface 714, which can be storedin user interface data and which can be used to configure and/or trainone or more machine-learned models that can be used to determine thestate of the user interface 714 and/or generate the simplified userinterface 716. For example, the user computing device 700 can use theuser's past interactions to determine the most frequently used userinteractions (e.g., the contacts to which messages are most often sentand/or the replies that are most frequently sent) or operations (e.g.,attaching an image to a text message) with the user interface 714 andcan then generate simplified user interface elements that are associatedwith the user's most frequently used interactions or operations with theuser interface 714.

The simplified user interface 716 can include the simplified userinterface element 722 (“CONFIRM”) that a user can use to confirm aselection within the user interface 714. For example, the simplifieduser interface element 722 can be used to confirm the addition ofanother contact (e.g., a contact to send a text message to) to the listof contacts for the messaging application. Further, the simplified userinterface 716 can include the simplified user interface element 724(“CANCEL”) that a user can use to undo or cancel an operation and/orselection within the user interface 714. For example, a user can use thesimplified user interface element 724 to undo a contact that wasincorrectly added to the list of contacts for the messaging application.The simplified user interface 716 can include the simplified userinterface element 726 (“PICK ME UP”) that a user can use to send a textmessage to a predetermined contact, with the text message requestingthat the predetermined contact pick the user up. For example, a user canuse the simplified user interface element (e.g., by touching thesimplified user interface element) to ask a predetermined relative orfriend to pick the user up from a location that can be determinedthrough use of a navigational component (not shown) of the usercomputing device 700. Further, the simplified user interface 716 caninclude the simplified user interface element 728 (“IN A MEETING”) thata user can use (e.g., by touching the simplified user interface element)to send a message to the sender of the most recent text message that wasreceived by the user. For example, during a meeting, a user can receivea message from a friend and can use the simplified user interfaceelement 728 to let that person know that the message was received andthat the user is currently in a meeting. In this manner, the simplifieduser interface 716 can facilitate the generation of a response to a textmessage that might otherwise involve many more time consuminginteractions including manually typing a response.

FIG. 8 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 800 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, the usercomputing device 300, and/or the user computing device 800. Furthermore,the user computing device 800 can perform one or more actions and/oroperations including the one or more actions and/or operations performedby the computing device 102, the computing system 130, the trainingcomputing system 150, and/or the user computing device 300.

As shown in FIG. 8, the user computing device 800 includes a displaycomponent 802, a user interface 814, a simplified user interface 816, aninterface element 818, an interface element 820, a simplified userinterface element 822, a simplified user interface element 824, and aninterface element 826.

In this example, the user computing device 800 is implementing aride-sharing application. The user interface 814 generates the mapinterface element 826 that is associated with a map of the geographicarea within a predetermined distance (e.g., three (3) kilometers) of theuser computing device 800; the interface element 818 which represents afirst vehicle (“VEHICLE 1”) that a user can select by touching (e.g.,pressing) the interface element 818; and the interface element 820 whichrepresents a second vehicle (“VEHICLE 2”) that a user can select bytouching the interface element 820.

The user computing device 800 can determine the state of the userinterface 814 and can generate the simplified user interface 816.Further, the user computing device 800 can use user interface dataassociated with the state of the user interface 814 to determine thestate of the user interface 814. For example, the user computing device800 can determine that the user interface 814 is the user interface of aride-sharing application and can further determine one or more locationsand/or one or more functionalities (e.g., displaying a map) of theinterface elements of the user interface 814 including the interfaceelement 818 and the interface element 820.

Further, based on the state of the user interface 814 the user computingdevice 800 can generate the simplified user interface 816. Thesimplified user interface can be based at least in part on a user'sprevious interactions with the user interface 814, which can be storedin user interface data and which can be used to configure and/or trainone or more machine-learned models that can be used to determine thestate of the user interface 814 and/or generate the simplified userinterface 816. For example, the user computing device 800 can use theuser's past interactions to determine the most frequently used userinteractions or operations with the user interface 814 and can thengenerate simplified user interface elements that are associated with theuser's most frequently used interactions or operations with the userinterface 814. For example, the user computing device 800 can determinethat the user frequently checks for vehicle pick-up notifications thatindicate the appearance of the vehicle and when the vehicle will beready to pick-up the user.

In some embodiments, the simplified user interface 816 can also includea summary of the one or more outputs associated with the user interface814. For example, the user interface data associated with the state ofthe user interface 814 can include the expected time of arrival of therequested vehicle and/or the type of the vehicle. The simplified userinterface 816 can include the simplified user interface element 822which is associated with an output of the simplified user interface 816.Upon determining the state of the user interface 814, and that thevehicle that the user selected will shortly be ready to pick up theuser, the user interface 814 can generate an audio output announcing,“YOUR VEHICLE, A RED SEDAN, WILL BE AT THE PICK-UP LOCATION IN 2MINUTES.” The user can cause the user computing device 800 to repeat themessage by tapping the simplified user interface element 822. Further,the simplified user interface 816 can include the simplified userinterface element 824 (“CANCEL”) that a user can use to undo or cancelan operation and/or selection within the user interface 814. Forexample, a user can use the simplified user interface element 824 tocancel the request for a vehicle.

FIG. 9 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 900 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300. Furthermore, the user computing device 900 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300.

As shown in FIG. 9, the user computing device 900 includes a displaycomponent 902, an imaging component 904, an audio input component 906, atactile component 908, a tactile component 910, an audio outputcomponent 912, a user interface 914, a simplified user interface 916, aninterface element 918, an interface element 920, a simplified userinterface element 922, a simplified user interface element 924, asimplified user interface element 926, a simplified user interfaceelement 928, and a plurality of interface elements 930.

The user computing device 900 can be configured to receive inputs (e.g.,information, data (e.g., application data and/or user interface data)and/or one or more user interactions). For example, the imagingcomponent 904 can include one or more cameras that can be configured toreceive input by capturing an image of an object that is placed in thefield of view of the imaging component 904. The audio input component906 can include one or more microphones that can be configured toreceive one or more sound inputs including vocal statements associatedwith an input to the simplified user interface 916. Further, the displaycomponent 902 can be configured to receive inputs to the user interface914 and/or the simplified user interface 916. For example, the displaycomponent 902 can be configured to receive inputs including touch inputsand other tactile interactions with the display component 902. Further,the display component 902 can be configured to receive inputs includingtouch inputs from one or more input devices (e.g., a mouse or a stylus).

Furthermore, the user computing device 900 can be configured to generateone or more outputs. For example, the audio output component 912 caninclude one or more loudspeakers and can be configured to generateaudible outputs including synthetic vocalizations. Further, the displaycomponent 902 can be configured to display a visual representation ofthe user interface 914 and/or the simplified user interface 916.

The user interface 914 and/or the simplified user interface 916 canreceive one or more inputs and/or generate one or more outputs. The userinterface 914 and/or the simplified user interface 916 can receiveinputs using any of the display component 902, the imaging component904, the audio input component 906, the tactile component 908, and/orthe tactile component 910. Further, the user interface 914 and/or thesimplified user interface 916 can generate outputs using any of thedisplay component 902, the tactile component 908, the tactile component910, and/or the audio output component 912.

As shown, the user computing device 900 includes a display component 902that is configured to display imagery (e.g., one or more images and/ortext) including the user interface 912 and/or the simplified userinterface 914, any combination of which can be associated with anapplication (e.g., an application that generates and/or uses the userinterface 914 and/or the simplified user interface 916).

In this example, the user computing device 900 is implementing aninterface configuration application that can be used to generate thesimplified user interface 916. For example, a user can select aninterface element of the plurality of interface elements 930 andassociate the interface element with a simplified interface element inthe simplified user interface element 916 by tapping the interfaceelement and then tapping a simplified user interface elements in thesimplified user interface element 916. For example, a user can selectthe interface element 918 (e.g., an interface element used to launch ane-mail application) and associate it with the simplified user interfaceelement 922; select the interface element 908 (e.g., an interfaceelement used to launch a voice dictation application) and associate itwith the simplified user interface element 924; select the interfaceelement 910 (e.g., an interface element used to toggle between usingvoice commands and using an onscreen keyboard to input text) andassociate it with the simplified user interface element 926; and selectthe interface element 920 (e.g., an interface element used to projectthe contents of the user interface 914 to another device) and associateit with the simplified user interface element 928.

In some embodiments, the configuration of the simplified user interface916 can be performed manually (e.g., a user selecting interface elementsand mapping those interface elements to the simplified user interface),automatically (e.g., the computing device 900 can use one or moremachine-learned models to generate the simplified user interface 916),or some combination of manual and automatic generation of the simplifieduser interface 916.

FIG. 10 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 1000 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300. Furthermore, the user computing device 1000 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300.

As shown in FIG. 10, the user computing device 1000 includes a displaycomponent 1002, an imaging component 1004, an audio input component1006, a tactile component 1008, a tactile component 1010, an audiooutput component 1012, a user interface 1014, a simplified userinterface 1016, an interface element 1018, an interface element 1020, asimplified user interface element 1022, a simplified user interfaceelement 1024, a simplified user interface element 1026, and a simplifieduser interface element 1028.

The user computing device 1000 can be configured to receive one or moreinputs including data (e.g., application data associated with a cameraapplication implemented on the user computing device 1000 and/or userinterface data) and/or one or more user interactions (e.g., the userperforming an interaction to capture an image) from a user of thecomputing device 1000). Further, the user computing device 1000 can beconfigured to generate the simplified user interface 1016 thatsimplifies one or more inputs and/or one or more outputs of the userinterface 1014. The user interface 1014 and/or the simplified userinterface 1016 can receive inputs and/or outputs via any of the displaycomponent 1002, the imaging component 1004, the audio input component1006, the tactile component 1008, and/or the tactile component 1010.Further, the user interface 1014 and/or the simplified user interface1016 can generate outputs using any of the display component 1002, thetactile component 1008, the tactile component 1010, and/or the audiooutput component 1012.

The display component 1002 can be configured to receive one or moreinputs and/or generate one or more outputs associated with the userinterface 1014 and/or the simplified user interface 1016. For example,the display component 1002 can be configured to receive inputs to theuser interface 1014 including touch inputs associated with capturing animage, capturing video, and/or recording slow-motion video. Further, thedisplay component 1002 can be configured to generate one or more outputsincluding the simplified user interface 1016, which can include asimplified version of the user interface 1014.

The imaging component 1004 can include one or more cameras that can beconfigured to receive input including one or more images that arecaptured when the interface element 1020 is touched by a user. The oneor more images received by the imaging component 1004 can be used in thegeneration of the simplified user interface 1016 by receiving visualinformation (e.g., an image of a user's eyes) that can be used as analternative to another input modality (e.g., a user pressing any of thetactile components 1008/1010). For example, a camera applicationimplemented on the user interface 1014 can receive touch inputs to thetactile component 1008 which can serve as a shutter control to capturean image. The simplified user interface 1016 can use the imagingcomponent 1004 to receive visual inputs of a user's head and can use oneor more image recognition techniques to determine that user's eyes haveblinked twice in a predetermined time period in order to initiatecapturing an image via the imaging component 1004.

The audio input component 1006 can include one or more microphones thatcan be configured to receive sound inputs including speech from a userof the user computing device 1000. The simplified user interface 1016generated on the user computing device 1000 can include a simplificationof visual interface elements (e.g., the interface element 1020 that isused to capture an image) by using the audio input component to receiveauditory inputs (e.g., the user speaking the command “CAPTURE PHOTO”) inorder to initiate capturing an image by the imaging component 1004.

The tactile components 1008/1010 can receive inputs including tactileinputs including touches and presses. The simplified user interface 1016generated on the user computing device 1000 can include a simplificationof visual interface elements (e.g., touching an interface element toscroll through a set of images) by using the tactile components1008/1010 to receive tactile inputs including pressing the tactilecomponent 1008 to show the next image in a set of images; and pressingthe tactile component 1010 to show the previous image in a set ofimages.

Furthermore, the user computing device 1000 can be configured togenerate outputs (e.g., visual outputs including the user interface 1014and/or the simplified user interface 1016) that are associated with thecamera application. The user computing device 1000 can simplify the userinterface 1014 by generating the simplified user interface 1016 thatincludes one or more outputs that have a different output modality fromthe output modality provided by the user interface 1014. For example,the user interface 1014 can generate visual outputs that display (e.g.,display on the display component 1002) one or more previously capturedimages and can use one or more image recognition techniques to show thename of a user specified face if the face is present in the previouslycaptured image that is being displayed. The simplified user interface1016 can use one or more image recognition techniques and hapticfeedback to generate a vibration when a particular face is present in animage that is being displayed.

In this example, the user computing device 1000 is implementing a cameraapplication (e.g., an application that can be used to capturephotographs and/or video via the imaging component 1004). The userinterface 1014 includes the interface element 1018 (“SLO-MO”) which canbe used to capture slow motion video via the imaging component 1004; andthe interface element 1020 that can be used to capture still images viathe imaging component 1004. The interface element 1018 and/or theinterface element 1020 can be respectively selected by a user touching(e.g., tapping) the interface element 1018 or the interface element1020.

The user computing device 1000 can determine the state of the userinterface 1014 and can generate the simplified user interface 1016.Further, the user computing device 1000 can use user interface dataassociated with the state of the user interface 1014 to determine thestate of the user interface 1014. For example, the user computing device1000 can determine that the user interface 1014 is the user interface ofa camera application and can further determine one or more locationsand/or one or more functionalities (e.g., capturing an image orrecording a video) of the interface elements of the user interface 1014including the interface element 1018 and the interface element 1020.

Further, based on the state of the user interface 1014, the usercomputing device 1000 can generate the simplified user interface 1016.The simplified user interface can be based at least in part on a user'sprevious interactions with the user interface 1014, which can be storedin user interface data and which can be used to configure and/or trainone or more machine-learned models that can be used to determine thestate of the user interface 1014 and/or generate the simplified userinterface 1016. For example, the user computing device 1000 can use theuser's past interactions (e.g., the camera modes that were used and/orthe individuals with whom images were previously shared) to determinethe most frequently used user interactions or operations with the userinterface 1014 and can then generate simplified user interface elementsthat are associated with the user's most frequently used interactions oroperations with the user interface 1014.

The simplified user interface 1016 can include the simplified userinterface element 1022 (“CONFIRM”) that a user can use to confirm anoperation (e.g., saving or deleting an image) within the user interface1014. For example, the simplified user interface element 1022 can beused to confirm an operation such as deleting an image that wascaptured. Further, the simplified user interface 1016 can include thesimplified user interface element 1024 (“CANCEL”) that a user can use toundo or cancel an operation and/or selection within the user interface1014. For example, a user can use the simplified user interface element1024 to undo the deletion of a recently deleted image. The simplifieduser interface element 1022 and the simplified user interface element1024 can provide a consistent set of controls within the user interface1014.

The simplified user interface 1016 can include the simplified userinterface element 1026 (“SHARE 1”) that a user can use to share an imagethat was captured with a predetermined contact by sending the image tothe predetermined contact. For example, a user can use the simplifieduser interface element 1026 to send an image to a friend or relative.The simplified user interface 1016 can include the simplified userinterface element 1028 (“SHARE 2”) that a user can use to share an imagethat was captured with a predetermined contact by sending the image tothe predetermined contact. For example, a user can use the simplifieduser interface element 1028 to send an image to a friend or relativethat is different from the friend or relative that is associated withthe simplified user interface element 1026.

FIG. 11 depicts an example of a user interface implemented on a usercomputing device according to example embodiments of the presentdisclosure. A user computing device 1100 can include one or moreattributes and/or capabilities of the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300. Furthermore, the user computing device 1100 canperform one or more actions and/or operations including the one or moreactions and/or operations performed by the computing device 102, thecomputing system 130, the training computing system 150, and/or the usercomputing device 300.

As shown in FIG. 11, the user computing device 1100 includes a displaycomponent 1102, an imaging component 1104, an audio input component1106, a tactile component 1108, an audio output component 1112, a userinterface 1114, a simplified user interface 1116, an interface element1118, an interface element 1120, a simplified user interface element1122, a simplified user interface element 1124, a simplified userinterface element 1126, and a simplified user interface element 1128.

The user computing device 1100 can be configured to receive one or moreinputs including data (e.g., application data associated with an e-mailapplication implemented on the user computing device 1100 and/or userinterface data) and/or one or more user interactions from a user of thecomputing device 1100). Further, the user computing device 1100 can beconfigured to generate the simplified user interface 1116 thatsimplifies one or more inputs and/or one or more outputs of the userinterface 1114. The user interface 1114 and/or the simplified userinterface 1116 can receive inputs and/or outputs via any of the displaycomponent 1102, the imaging component 1104, the audio input component1106, the tactile component 1108, and/or the tactile component 1110.Further, the user interface 1114 and/or the simplified user interface1116 can generate outputs using any of the display component 1102, thetactile component 1108, the tactile component 1110, and/or the audiooutput component 1112.

The display component 1102 can be configured to receive one or moreinputs and/or generate one or more outputs associated with the userinterface 1114 and/or the simplified user interface 1116. For example,the display component 1102 can be configured to receive inputs to theuser interface 1114 including touch inputs associated with composing ane-mail, sending an e-mail, and/or accessing a message in the inbox.Further, the display component 1102 can be configured to generate one ormore outputs including the simplified user interface 1116, which caninclude a simplified version of the one or more interface elements ofthe user interface 1114.

The imaging component 1104 can include one or more cameras that can beconfigured to receive input including one or more images. The one ormore images received by the imaging component 1104 can be used in thegeneration of the simplified user interface 1116 by receiving visualinformation (e.g., an of the user including a user's hand) that can beused as an alternative to another input modality (e.g., a user touchingthe display component 1102). For example, an e-mail applicationimplemented on the user interface 1114 can receive touch inputs to thedisplay component 1102 as the default input to advance to the nextmessage in a list of messages in the e-mail inbox. The simplified userinterface 1116 can use the imaging component 1104 to receive visualinputs of a user's hand and can use one or more image recognitiontechniques to determine that a hand waving from left to right willadvance to the next message in the list of messages in the e-mail inbox.

The audio input component 1106 can include one or more microphones thatcan be configured to receive sound inputs including speech from a userof the user computing device 1100. The simplified user interface 1116generated on the user computing device 1100 can include a simplificationof visual interface elements (e.g., touching the display component 1104to advance to the next message in a list of messages in the e-mailinbox) by using the audio input component to receive auditory inputs(e.g., a user saying “NEXT MESSAGE” aloud) to advance to the nextmessage in a list of messages in the e-mail inbox.

The tactile components 1108/1110 can receive inputs including tactileinputs including touches and presses. The simplified user interface 1116generated on the user computing device 1100 can include a simplificationof visual interface elements (e.g., touching one of the plurality ofinterface elements 1118 to select an e-mail message and then pressinganother interface element to delete the message) by using the tactilecomponents 1108/1110 to delete a selected e-mail message.

Furthermore, the user computing device 1100 can be configured togenerate outputs (e.g., visual outputs including the user interface 1114and/or the simplified user interface 1116) that are associated with thee-mail application. The user computing device 1100 can simplify the userinterface 1114 by generating the simplified user interface 1116 thatincludes one or more outputs that have a different output modality fromthe output modality provided by the user interface 1114. For example,the user interface 1114 can generate visual outputs that display thecontents of an e-mail message on the display component 1102. Thesimplified user interface 1116 can use the audio output component 1112(e.g., one or more loudspeakers) to generate audible outputs that use asynthetic voice to read aloud the contents of an e-mail message.

In this example, the user computing device 1100 is implementing ane-mail application. The user interface 1114 includes the plurality ofinterface elements 1118 which represent interface elements that a usercan interact with to open a respective message (a user tapping “MESSAGE1” will open the message associated with “MESSAGE 1”) which can bedisplayed within the interface element 1110 (“MESSAGE CONTENTS”); andthe interface element 1120 (“COMPOSE”) that can be used to compose ane-mail message. The interface element 1118 and/or the interface element1120 can be respectively selected by a user touching (e.g., tapping) theinterface element 1118 or the interface element 1120.

The user computing device 1100 can determine the state of the userinterface 1114 and can generate the simplified user interface 1116.Further, the user computing device 1100 can use user interface dataassociated with the state of the user interface 1114 to determine thestate of the user interface 1114. For example, the user computing device1100 can determine that the user interface 1114 is the user interface ofan e-mail application and can further determine one or more locationsand/or one or more functionalities (e.g., checking a message and/orcomposing a message) of the interface elements of the user interface1114 including the interface element 1118 and the interface element1120.

Further, based on the state of the user interface 1114 the usercomputing device 1100 can generate the simplified user interface 1116.The simplified user interface can be based at least in part on a user'sprevious interactions with the user interface 1114, which can be storedin user interface data and which can be used to configure and/or trainone or more machine-learned models that can be used to determine thestate of the user interface 1114 and/or generate the simplified userinterface 1116. For example, the user computing device 1100 can use theuser's past interactions to determine the most frequently used userinteractions or operations with the user interface 1114 and can thengenerate simplified user interface elements that are associated with theuser's most frequently used interactions or operations with the userinterface 1114.

The simplified user interface 1116 can include the simplified userinterface element 1122 (“CONFIRM”) that a user can use to confirm aselection within the user interface 1114. For example, the simplifieduser interface element 1122 can be used to confirm the addition ofanother contact (e.g., a contact to send an e-mail message to) to thelist of contacts for the e-mail application. Further, the simplifieduser interface 1116 can include the simplified user interface element1124 (“CANCEL”) that a user can use to undo or cancel an operationand/or selection within the user interface 1114. For example, a user canuse the simplified user interface element 1124 to undo a contact thatwas incorrectly added to the list of contacts for the messagingapplication. The simplified user interface 1116 can include thesimplified user interface element 1126 (“CC: TEAM 1”) that a user canuse to send an e-mail that the user composed to a predetermined group ofe-mail recipients. For example, a user can use the simplified userinterface element 1126 to send an e-mail to a specific group ofindividuals at the user's place of business. The simplified userinterface 1116 can include the simplified user interface element 1128(“CC: TEAM 2”) that a user can use to send an e-mail that the usercomposed to a predetermined group of e-mail recipients. For example, auser can use the simplified user interface element 1128 to send ane-mail to a specific group of individuals at the user's place ofbusiness that is different from the group of individuals associated withthe simplified user interface element 1126.

FIG. 12 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure. One or moreportions of the method 1200 can be executed and/or implemented on one ormore computing devices or computing systems including, for example, thecomputing device 102, the computing system 130, the training computingsystem 150, and/or the user computing device 300. Further, one or moreportions of the method 1200 can be executed or implemented as analgorithm on the hardware devices or systems disclosed herein. FIG. 12depicts steps performed in a particular order for purposes ofillustration and discussion. Those of ordinary skill in the art, usingthe disclosures provided herein, will understand that various steps ofany of the methods disclosed herein can be adapted, modified,rearranged, omitted, and/or expanded without deviating from the scope ofthe present disclosure.

At 1202, the method 1200 can include accessing data which can includeuser interface data. The user interface data can include one or morestates of a user interface that includes one or more interface elements(e.g., interface elements that a user can interact with to perform oneor more operations associated with an application). For example, thecomputing device 102 can be smartphone device that is configured toaccess user interface data that is associated with one or moreapplications (e.g., a mapping application) that are implemented on thecomputing device 102. The user interface data can be accessed locally onthe smartphone device (e.g., user interface data stored on a memorydevice of the smartphone). In some embodiments, the user interface datacan be at a remote location that is accessed by the smartphone (e.g.,accessed via a wireless and/or wired network).

At 1204, the method 1200 can include determining a state of the userinterface. The state of the user interface can be determined based atleast in part on the user interface data. Further, the state of the userinterface can include one or more functionalities of the one or moreinterface elements. For example, the computing device 102 can use theuser interface data to determine the state of the user interface byperforming one or more operations including determining one or morestates and/or one or more functionalities of the user interface and/orthe one or more interface elements. Determining the one or morefunctionalities of the user interface can include determining the one ormore operations that the one or more interface elements and/or the userinterface can perform, the one or more applications associated with theone or more interface elements and/or the user interface, and/or the oneor more inputs and/or one or more outputs associated with the one ormore interface elements.

Further, the computing device can determine one or more inputs and/orthe one or more outputs that are associated with the user interface. Forexample, the computing device 102 can determine one or more interfaceelements of the user interface including: one or more input elementsincluding controls that can activate, operate, and/or navigateapplications and/or the user interface; and one or more output elementsto provide output of the user interface including displaying images,producing sound, and/or producing haptic feedback. Further, thecomputing device 102 can determine one or more relationships between theone or more interface elements including the relative position of theone or more interface elements, and/or the ways in which the one or moreinterface elements can communicate and interact with one another andwith one or more applications. For example, the computing device 102 candetermine the state of a user interface associated with a telephoneapplication. The state of the user interface can include one or morefunctionalities and/or one or more locations of the numeric keypadcontrols, the call control, and the hang-up control. In someembodiments, determination of the state of the user interface caninclude the use of one or more machine-learning models that have beenconfigured and/or trained to receive an input including the userinterface data (e.g., user interface data including an image of the userinterface, a node hierarchy of the user interface, and/or informationassociated with the one or more functionalities of the user interface),perform one or more operations on the input, and generate an outputincluding the state of the user interface.

At 1206, the method 1200 can include mapping the one or more interfaceelements to one or more simplified user interface elements. Mapping theone or more interface elements to the one or more simplified userinterface elements can be based at least in part on the one or morefunctionalities of the one or more interface elements. For example, thecomputing device 102 can map the one or more interface elements to oneor more simplified user interface elements by comparing the one or moreinterface elements to the one or more simplified user interface elementsand determining the one or more interface elements that match or aresimilar to one or more attributes of the one or more simplified userinterface elements. The one or more interface elements that satisfy oneor more criteria associated with the extent (e.g., the number ofattributes that match, which types of attributes match, and/or aweighting between the number and/or types of attributes that match) towhich the one or more interface elements match or are similar to the oneor more simplified user interface elements can be mapped to the one ormore simplified user interface elements.

At 1208, the method 1200 can include generating a simplified userinterface. Generating the simplified user interface can be based atleast in part on the one or more simplified user interface elements. Anyof the one or more simplified user interface elements can be configuredto perform one or more operations which can include one or moreoperations associated with the one or more functionalities of the one ormore interface elements. For example, the computing device 102 canperform one or more operations to implement the simplified userinterface, including rendering one or more images of the one or moresimplified user interface elements on a display device (e.g., generatinga graphical user interface that includes a simplified user interface asan overlay on the user interface), generating audible output includingsynthetic voice instructions (e.g., generating instructions that requesta vocal command from a user) based on the one or more user interfaceelements, and/or illuminating a portion of the user interface toindicate that it can receive input (e.g., illuminating a button with redlighting to indicate that the button can be pressed to end a phonecall).

At 1210, the method 1200 can include determining one or more proposedsimplified user interface elements for each of the one or more interfaceelements that do not match the one or more simplified user interfaceelements. For example, the computing device 102 can determine the one ormore interface elements with one or more attributes that do not match apredetermined amount (e.g., a predetermined number of the one or moreattributes) or type of one or more attributes of the one or moresimplified user interface elements. The one or more interface elementsthat do not match a predetermined amount or type of attributes of theone or more simplified user interface elements can be associated withone or more proposed simplified user interface elements that are withina predetermined range of similarity. For example, a match can bedetermined not to have occurred when less than seventy-five (75) percentof the one or more attributes of the one or more interface elements andthe one or more proposed simplified user interface elements match. Byway of further example, the computing device 102 can include one or moremachine-learning models that are configured and/or trained to determinethe one or more proposed simplified user interface elements based atleast in part on the one or more one or more interface elements that donot match a predetermined amount or type of attributes of the one ormore simplified user interface elements.

In some embodiments, the one or more proposed simplified user interfaceelements can include one or more simplified user interface elements thatwere not mapped to the one or more simplified user interface elements.Further, the one or more simplified user interface elements canrespectively correspond to the one or more proposed simplified userinterface elements that were not mapped to the one or more simplifieduser interface elements.

At 1212, the method 1200 can include generating a prompt. The prompt caninclude a request. The request can include a request for a user toselect at least one of the one or more proposed simplified userinterface elements. For example, the computing device 102 can generate avisual prompt on a display associated with the user interface and/or anauditory prompt via an audio output associated with the user interface.The prompt can list the one or more proposed simplified user interfaceelements, each of which can be associated with a respectivecorresponding one or more interface elements. For example, the promptcan request the user to “PLEASE INDICATE THE SIMPLIFIED USER INTERFACEELEMENTS THAT CORRECTLY MATCH THE INTERFACE ELEMENTS OF THE USERINTERFACE.” In some embodiments, the one or more proposed simplifieduser interface elements selected by the user and/or the one or moreproposed simplified user interface elements not selected by the user canbe used to configure and/or train one or more machine-learning models todetermine the one or more proposed simplified user interface elementsand/or to map the one or more interface elements to the one or moresimplified user interface elements.

FIG. 13 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure. One or moreportions of the method 1300 can be executed and/or implemented on one ormore computing devices or computing systems including, for example, thecomputing device 102, the computing system 130, the training computingsystem 150, and/or the user computing device 300. Further, one or moreportions of the method 1300 can be executed or implemented as analgorithm on the hardware devices or systems disclosed herein. In someembodiments, one or more portions of the method 1300 can be performed aspart of the method 1200 that is depicted in FIG. 12. FIG. 13 depictssteps performed in a particular order for purposes of illustration anddiscussion. Those of ordinary skill in the art, using the disclosuresprovided herein, will understand that various steps of any of themethods disclosed herein can be adapted, modified, rearranged, omitted,and/or expanded without deviating from the scope of the presentdisclosure.

At 1302, the method 1300 can include determining the state of the userinterface based at least in part on one or more features of the userinterface. The computing device 102 can perform one or more operationsto analyze one or more visual features of the user interface includingone or more locations of the one or more interface elements and therespective appearance of the one or more interface elements. Forexample, analysis of the one or more visual features can be based atleast in part on analysis of one or more screen captures of the userinterface. The computing device 102 can then determine the state of theuser interface based at least in part on the one or more visual featuresof the one or more interface elements.

In some embodiments, determining the state of the user interface caninclude generating an input that can be provided to one or moremachine-learned models that are configured and/or trained to determineone or more states of the user interface.

At 1304, the method 1300 can include determining the state of the userinterface based at least in part on the user interface data and/or onemore machine-learned models. The one or more machine-learned models canbe configured to classify the one or more interface elements into one ormore classes (e.g., semantic classes) respectively. For example, theuser interface data including one or more visual features of the userinterface can be provided as an input to the one or more machine-learnedmodels which can be configured and/or trained to associate the one ormore visual features of the user interface with respective simplifieduser interface element of the one or more simplified user interfaceelements.

FIG. 14 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure. One or moreportions of the method 1400 can be executed and/or implemented on one ormore computing devices or computing systems including, for example, thecomputing device 102, the computing system 130, the training computingsystem 150, and/or the user computing device 300. Further, one or moreportions of the method 1400 can be executed or implemented as analgorithm on the hardware devices or systems disclosed herein. In someembodiments, one or more portions of the method 1400 can be performed aspart of the method 1200 that is depicted in FIG. 12. FIG. 14 depictssteps performed in a particular order for purposes of illustration anddiscussion. Those of ordinary skill in the art, using the disclosuresprovided herein, will understand that various steps of any of themethods disclosed herein can be adapted, modified, rearranged, omitted,and/or expanded without deviating from the scope of the presentdisclosure.

At 1402, the method 1400 can include comparing the tree hierarchy to oneor more stored tree hierarchies associated with one or more storedinterface elements. For example, the computing device 102 can performone or more tree hierarchy or node hierarchy comparison operationsincluding comparing the types of nodes (e.g., the one or morefunctionalities associated with each node in the tree hierarchy) in thetree hierarchy to one or more stored tree hierarchies, comparing theconnections (e.g., which other nodes is a node connected to) betweennodes in the tree hierarchy to the connections between nodes in the oneor more stored tree hierarchies, and/or comparing the size of the treehierarchy (e.g., the number of nodes in a tree hierarchy and/or thedistance from the top leaf of the tree hierarchy to the furthest bottomleaf of the tree hierarchy) to the to the sizes of the one or morestored tree hierarchies.

At 1404, the method 1400 can include determining whether or if one ormore portions of the tree hierarchy match or are similar to one or moreportions of the one or more stored tree hierarchies. The computingdevice 102 can determine whether the tree hierarchy matches or issimilar to the one or more portions of the one or more stored treehierarchies based at least in part on the results of the one or moretree hierarchy comparison operations. Further, a greater extent ofmatching and/or similarity can be associated with a greater number ofsimilar types of nodes between the one or more portions of the treehierarchy match and one or more portions of the one or more stored treehierarchies, a greater number of connections in the one or more portionsof the tree hierarchy that match one or more portions of the one or morestored tree hierarchies, and/or a greater similarity in the size of theone or more portions of the tree hierarchy match compared to the one ormore portions of the one or more stored tree hierarchies.

At 1406, the method 1400 can include determining one or moreprobabilities or likelihoods of each of the one or more interfaceelements receiving a user interaction. For example, the computing device102 after determining the one or more functionalities of the one or moreinterface elements and/or the context of the user interface (e.g., theapplication that is operating, the sub-set of the user interface withinwhich an interface element is being used), the computing device 102 canaccess interaction probability data that includes the probability of theone or more interface elements receiving a user interaction (e.g., auser touching an interface element that controls an operation of anapplication).

At 1408, the method 1400 can include mapping a predetermined number ofthe one or more simplified user interface elements to a respectivenumber of the one or more interface elements with a probability ofinteraction that satisfies one or more interaction criteria. Forexample, the computing device 102 can map the one or more simplifieduser interface elements to the one or more interface elements that havethe highest probability of receiving a user interaction and/or map theone or more simplified user interface elements to the one or moreinterface elements that satisfy (e.g., exceed) a threshold probability.For example, the computing device 102 can determine that the simplifieduser interface can accommodate four simplified user interface elementsand can map the four interface elements with the highest probabilitiesof interaction to the four simplified user interface elements.

At 1410, the method 1400 can include mapping the one or more interfaceelements to the one or more simplified user interface elements based atleast in part on one or more machine-learned models. The one or moremachine-learned models are configured to determine one or more featuresof the one or more interface elements that match the one or moresimplified user interface elements. For example, the user computingdevice 102 can provide input including the user interface data and/orthe state of the user interface to one or more machine-learned modelsthat are configured and/or trained to map the one or more interfaceelements to one or more simplified user interface elements.

FIG. 15 depicts a flow diagram of simplified user interface generationaccording to example embodiments of the present disclosure. One or moreportions of the method 1500 can be executed and/or implemented on one ormore computing devices or computing systems including, for example, thecomputing device 102, the computing system 130, the training computingsystem 150, and/or the user computing device 300. Further, one or moreportions of the method 1500 can be executed or implemented as analgorithm on the hardware devices or systems disclosed herein. In someembodiments, one or more portions of the method 1500 can be performed aspart of the method 1200 that is depicted in FIG. 12. FIG. 15 depictssteps performed in a particular order for purposes of illustration anddiscussion. Those of ordinary skill in the art, using the disclosuresprovided herein, will understand that various steps of any of themethods disclosed herein can be adapted, modified, rearranged, omitted,and/or expanded without deviating from the scope of the presentdisclosure.

At 1502, the method 1500 can include determining, based at least in parton the user interface data, one or more locations of the one or moreinterface elements respectively. For example, the computing device 102can use the user interface data (which can include informationassociated with the one or more locations of the one or more interfaceelements) to determine one or more relative locations (e.g., relative toone or more other interface elements), and/or one or more absolutelocations (e.g., absolute with respect to the user interface) of the oneor more interface elements.

At 1504, the method 1500 can include generating the simplified userinterface in a portion of the user interface in which the fewest of theone or more interface elements are located. For example, the computingdevice 102 can determine that the bottom quarter portion of the userinterface does not have any interface elements and that the top quarterportion of the user interface has four interface elements. The computingdevice 102 can generate the simplified user interface in the bottomquarter portion of the user interface.

At 1506, the method 1500 can include determining, based at least in parton the user interface data, the one or more interface elements thatshould not be obstructed. The computing device 102 can use the userinterface data to determine the one or more interface elements that areassociated with one or more functionalities that may be deemed to benecessary (e.g., an interface element that is used to exit or close anapplication) for proper operation of the application.

At 1508, the method 1500 can include generating the simplified userinterface in one or more portions of the user interface that do notinclude the one or more interface elements that should not beobstructed. The computing device 102 can generate the simplified userinterface in a portion of the user interface that does not include aninterface element that is used to close or exit the application that iscurrently being used.

At 1510, the method 1500 can include generating one or more indicationsto identify the one or more interface elements that are mapped to theone or more simplified user interface elements. For example, thecomputing device 102 can render a green border around the one or moreinterface elements that are mapped to the one or more simplified userinterface elements. Further, the computing device 102 can generate ayellow highlight around the one or more interface elements that aremapped to the one or more simplified user interface elements.

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations and/or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. For instance, features illustrated or described aspart of one embodiment can be used with another embodiment to yield astill further embodiment. Thus, it is intended that the presentdisclosure cover such alterations, variations, and equivalents.

What is claimed is:
 1. A computer-implemented method of user interfacegeneration, the computer-implemented method comprising: accessing, by auser computing device associated with a user and comprising one or moreprocessors, user interface data comprising one or more states of a userinterface comprising one or more interface elements; determining, by theuser computing device, a state of the user interface based at least inpart on the user interface data, wherein the state of the user interfacecomprises one or more functionalities of the one or more interfaceelements; mapping, by the user computing device, the one or moreinterface elements to one or more simplified user interface elementsbased at least in part on the one or more functionalities of the one ormore interface elements, wherein the one or more interface elements aregrouped into one or more element classes respectively mapped to the oneor more simplified user interface elements, and wherein the mapping isconfigured for representing one or more members of a respective elementclass with a respective simplified user interface element correspondingto the respective element class; and generating, by the user computingdevice, a simplified user interface based at least in part on the one ormore simplified user interface elements, wherein the one or moresimplified user interface elements are configured to perform one or moreoperations associated with the one or more functionalities of the one ormore interface elements.
 2. The computer-implemented method of claim 1,wherein the user interface is configured to receive one or moretask-directed inputs or one or more task-auxiliary inputs, and whereinthe user interface is configured to generate one or more task-directedoutputs or one or more task-auxiliary outputs, wherein the one or moretask-directed inputs or the one or more task-directed outputs areassociated with one or more tasks that are extrinsic to user interactionwith the user interface, and wherein the one or more task-auxiliaryinputs or the one or more task-auxiliary outputs are associated with oneor more tasks that are not extrinsic to user interaction with the userinterface.
 3. The computer-implemented method of claim 1, wherein thedetermining, by the user computing device, a state of the user interfacebased at least in part on the user interface data comprises:determining, by the user computing device, the state of the userinterface based at least in part on one or more features of the userinterface, wherein the one or more features of the user interfacecomprises one or more text labels associated with the one or moreinterface elements, a visual appearance of the one or more interfaceelements, one or more sizes of the one or more interface elements, oneor more locations of the one or more interface elements, one or moreinput modalities of the one or more interface elements, or one or moreoutput modalities of the one or more interface elements.
 4. Thecomputer-implemented method of claim 1, wherein the determining, by theuser computing device, a state of the user interface based at least inpart on the user interface data, wherein the state of the user interfacecomprises one or more functionalities of the one or more interfaceelements comprises: determining, by the user computing device, the stateof the user interface based at least in part on the user interface dataand one or more machine-learned models, wherein the one or moremachine-learned models are configured to classify the one or moreinterface elements into one or more semantic classes respectively, andwherein each of the one or more semantic classes is associated with arespective simplified user interface element of the one or moresimplified user interface elements.
 5. The computer-implemented methodof claim 4, wherein the one or more interface elements or the one ormore simplified user interface elements are clustered into the one ormore semantic classes based at least in part on one or more features ofthe one or more interface elements.
 6. The computer-implemented methodof claim 1, wherein the user interface data comprises a tree hierarchyassociated with the one or more interface elements, and wherein themapping, by the user computing device, the one or more interfaceelements to one or more simplified user interface elements based atleast in part on the one or more functionalities of the one or moreinterface elements comprises: comparing, by the user computing device,the tree hierarchy to one or more stored tree hierarchies associatedwith one or more stored interface elements; and determining, by the usercomputing device, whether one or more portions of the tree hierarchymatch one or more portions of the one or more stored tree hierarchies.7. The computer-implemented method of claim 6, wherein the treehierarchy is associated with an application programming interface (API)that defines one or more attributes of the one or more interfaceelements, and wherein the one or more attributes comprise one or moreattributes associated with labels on the one or more interface elements,one or more attributes associated with accessibility markup on the oneor more interface elements, attributes associated with a layout of theone or more interface elements, attributes associated with one or morerespective inputs of the one or more interface elements, attributesassociated with one or more respective outputs of the one or moreinterface elements, or attributes associated with an appearance of theone or more interface elements.
 8. The computer-implemented method ofclaim 1, wherein the mapping, by the user computing device, the one ormore interface elements to one or more simplified user interfaceelements based at least in part on the one or more functionalities ofthe one or more interface elements comprises: determining, by the usercomputing device, one or more probabilities of each of the one or moreinterface elements receiving a user interaction; and mapping, by theuser computing device, a predetermined number of the one or moresimplified user interface elements to a respective number of the one ormore interface elements that satisfy a threshold probability ofreceiving a user interaction.
 9. The computer-implemented method ofclaim 1, wherein the mapping, by the user computing device, the one ormore interface elements to one or more simplified user interfaceelements based at least in part on the one or more functionalities ofthe one or more interface elements comprises: mapping, by the usercomputing device, the one or more interface elements to the one or moresimplified user interface elements based at least in part on one or moremachine-learned models, wherein the one or more machine-learned modelsare configured to determine one or more features of the one or moreinterface elements that match the one or more simplified user interfaceelements.
 10. The computer-implemented method of claim 1, furthercomprising: determining, by the user computing device, one or moreproposed simplified user interface elements for each of the one or moreinterface elements that do not match the one or more simplified userinterface elements; and generating, by the user computing device, aprompt comprising a request for a user to select at least one of the oneor more proposed simplified user interface elements.
 11. Thecomputer-implemented method of claim 1, wherein the generating, by theuser computing device, a simplified user interface based at least inpart on the one or more simplified user interface elements, wherein theone or more simplified user interface elements are configured to performone or more operations associated with the one or more functionalitiesof the one or more interface elements comprises: determining, by theuser computing device, based at least in part on the user interfacedata, one or more locations of the one or more interface elementsrespectively; and generating, by the user computing device, thesimplified user interface in a portion of the user interface in whichthe fewest of the one or more interface elements are located.
 12. Thecomputer-implemented method of claim 1, wherein the generating, by theuser computing device, a simplified user interface based at least inpart on the one or more simplified user interface elements, wherein theone or more simplified user interface elements are configured to performone or more operations associated with the one or more functionalitiesof the one or more interface elements comprises: determining, by theuser computing device, based at least in part on the user interfacedata, the one or more interface elements that should not be obstructed;and generating, by the user computing device, the simplified userinterface in one or more portions of the user interface that do notinclude the one or more interface elements that should not beobstructed.
 13. The computer-implemented method of claim 1, furthercomprising: generating, by the user computing device, one or moreindications to identify the one or more interface elements that aremapped to the one or more simplified user interface elements, whereinthe one or more indications comprise one or more highlights of the oneor more interface elements, one or more color changes of the one or moreinterface elements, one or more changes to a brightness of the one ormore interface elements, one or more markings on the one or moreinterface elements, or one or more shapes around the one or moreinterface elements.
 14. The computer-implemented method of claim 1,wherein the simplified user interface is presented in a separate regionor device from the user interface or the simplified user interface issuperimposed over a portion of the user interface.
 15. One or moretangible non-transitory computer-readable media storingcomputer-readable instructions that when executed by one or moreprocessors cause the one or more processors to perform operations, theoperations comprising: accessing user interface data comprising one ormore states of a user interface comprising one or more interfaceelements; determining a state of the user interface based at least inpart on the user interface data, wherein the state of the user interfacecomprises one or more functionalities of the one or more interfaceelements; mapping the one or more interface elements to one or moresimplified user interface elements based at least in part on the one ormore functionalities of the one or more interface elements, wherein theone or more interface elements are grouped into one or more elementclasses respectively mapped to the one or more simplified user interfaceelements, and wherein the mapping is configured for representing one ormore members of a respective element class with a respective simplifieduser interface element corresponding to the respective element class;and generating a simplified user interface based at least in part on theone or more simplified user interface elements, wherein the one or moresimplified user interface elements are configured to perform one or moreoperations associated with the one or more functionalities of the one ormore interface elements.
 16. The one or more tangible non-transitorycomputer-readable media of claim 15, wherein the simplified userinterface is associated with control of an application, and wherein theone or more simplified user interface elements comprise one or morepredetermined application controls or a control to remove the simplifieduser interface.
 17. The one or more tangible non-transitorycomputer-readable media of claim 16, wherein the application comprises atext messaging application, an e-mail application, or a telephoneapplication, and wherein the one or more predetermined applicationcontrols comprise one or more message templates, one or morepredetermined geographic locations, or one or more predetermined usercontacts.
 18. A computing system comprising: one or more processors; oneor more non-transitory computer-readable media storing instructions thatwhen executed by the one or more processors cause the one or moreprocessors to perform operations, the operations comprising: accessinguser interface data comprising one or more states of a user interfacecomprising one or more interface elements; determining a state of theuser interface based at least in part on the user interface data,wherein the state of the user interface comprises one or morefunctionalities of the one or more interface elements; mapping the oneor more interface elements to one or more simplified user interfaceelements based at least in part on the one or more functionalities ofthe one or more interface elements, wherein the one or more interfaceelements are grouped into one or more element classes respectivelymapped to the one or more simplified user interface elements, andwherein the mapping is configured for representing one or more membersof a respective element class with a respective simplified userinterface element corresponding to the respective element class; andgenerating a simplified user interface based at least in part on the oneor more simplified user interface elements, wherein the one or moresimplified user interface elements are configured to perform one or moreoperations associated with the one or more functionalities of the one ormore interface elements.
 19. The computing system of claim 18, whereinthe simplified user interface is associated with an input modality thatis different from that of the user interface, and wherein the inputmodality of the simplified user interface comprises a touch inputmodality, an audio input modality, or a gesture input modality.
 20. Thecomputing system of claim 18, wherein the simplified user interface isassociated with an output modality that is different from that of theuser interface, and wherein the output modality of the simplified userinterface comprises a visual output modality, an auditory outputmodality, or a tactile output modality.